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Noninvasive Detection of the Brain’s Magnetic Fields 
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One SQUID Device 


The Office of Naval Research has been supporting 
superconducting research for measuring magnetic fields 
since the mid-1950’s. The early research was directed to- 
wards devices and techniques for possible submarine detec- 
tion. Later, this research was applied to detecting the mag- 
netic fields eminating from the human body. The first 
measurements made in 1969 by ONR supported scientists 
were of the magnetic fields associated with the human heart; 
an event which opened a new era for biomagnetic research. 
Over the years, ONR support has led the way in developing 


supersensitive devices for measuring the magnetic fields of 
the brain. Professor S. L. Williamson and L. Kaufman of 
New York University detected magnetic field activity in the 
brain with a one SQUID (Superconducting Quantum Inter- 
ference Device) gradiometer in an unshielded room in 1974. 
Recently, Williamson and Kaufman developed FREDDY, a 
multisensor SQUID gradiometer. FREDDY can map the 
circuitry of the brain and may produce measurements which 
may lead in the future to an understanding of how brain cells 
communicate. 
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Introduction 


A great deal of structural and biechemical information 
about the nervous system has accumulated within the last 
twenty-five years. Although all the details of the neu- 
roanatomic and neurophysiologic mechanisms have not 
been specified, many scientists believe that our understand- 
ing of information flow through neural circuits is advanced 
enough to begin a serious attempt to explain complex cogni- 
tive processes such as learning and memory in biological 
terms. We are at the point were realistic and powerful mod- 
els of cellular “information processing” networks underly- 
ing learning and memory can be developed. It is one of the 
underlying assumptions of the Physiology/Neurobiology 
Group at the Office of Naval Research (ONR) that increased 
understanding of the “original” information processing sys- 
tem, the brain, will lead to advances in the design of infor- 
mation processing systems. Entirely new approaches to 
large scale information storage, retrieval and decision mak- 
ing systems may well be the result. 

The three articles in this issue of Naval Research Re- 
views have been written by contractors representing pro- 
grams in the current ONR Learning and Memory Program. 
The program was begun in 1982 and will continue through 
1987. Although acknowledged to be at the forefront in their 
respective fields, the investigators who participate in this 
program have all made a commitment to broaden their ex- 
perimental expertise and to begin to investigate learning and 
memory with a more multidisciplinary approach. 

Dr. Richard Thompson’s contribution deals with the 
identification, localization, and analysis of essential mem- 
ory trace circuits in the mammalian brain. He has taken a 
very discrete, highly reproducible type of behavior (that has 
been described in most mammals, including humans) and 
traced the brain circuits that acquire and store the informa- 
tion needed to reproduce the behavior. Dr. Thompson’s 
we... is primarily empirical but is strongly influenced by 
theoretical computational modelling. His article reaffirms 
the importance of the cerebellum in the learning and mem- 
ory process and suggests that memory storage may be truly 
localized. 

Dr. James McGaugh leads a group that has approached 
the problem of understanding the neurobiology of learning 
and memory in different yet complementary ways. 
Dr. McGaugh’s group has long been a leading force in in- 
vestigating the chemistry of the brain and peripheral hor- 
monal systems which facilitate and impair memory storage. 
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Dr. Gary Lynch’s work has centered around the identifica- 
tion of an enzymatic process that may be the critical step in 
forming certain types of memory. Dr. Norman 
Weinberger’s interests include how specific cells in the au- 
ditory system exhibit learning-like behavior prior to the 
appearance of behavioral manifestations of memory. Lastly, 
Dr. Richard Granger’s group has developed an innovative 
and powerful process model of learning and memory as the 
basis for artificial intelligence approaches to the field. These 
four authors demonstrate that these issues are highly interre- 
lated, and that findings at one level of analysis are necessary 
for advances at other levels. 

Dr. Stephen Kosslyn’s contribution to this issue is rep- 
resentative of a more cognitive approach to learning and 
memory, one that Dr. Kosslyn describes as “computational 
neuropsychology.” Much of this work is directed at a gen- 
eral theory of high level vision. This theory consists of those 
aspects of visual processing where one’s knowledge and 
beliefs are utilized in carrying out information processing. 
This may include aspects of recognition, navigation, track- 
ing, and mental imagery. Some of the empirical data for this 
work are derived from “split-brain” human subjects whose 
unique lateralized condition allows the study of human vi- 
sual representation in two separate cerebral hemispheres. 
Dr. Kosslyn’s theory of high level vision makes use of a 
hybrid functional architecture, one component consisting of 
“connectionist” parallel processing networks and one com- 
ponent consisting of a more standard von Neumann-like 
architecture. This work, like that of the other contributors to 
this issue, may have profound implications for a new gener- 
ation of artificial-intelligence devices. 

The human brain and its functional abilities represent 
the ultimate in evolutionary processes that began millions of 
years ago. Understanding how learning and memory takes 
place is one of the most important basic questions still to be 
answered in the life sciences. I hope this issue of Naval 
Research Reviews conveys some of the current excitiment 
in this field. 


JOEL L. DAVIS 

Program Manager 
Physiology/Neurobiology Group 
Biological Sciences Division, ONR 





LOCALIZATION 
OF 
MEMORY 
TRACES 
IN 
THE 
MAMMALIAN 
BRAIN 


Richard F. Thompson, 
Stanford University 


Perhaps the ultimate goal of research in the area of 
brain function and behavior is an understanding of the 
human mind. A key aspect of the mind is the nature of our 
thoughts or cognitions—how are they assembled and main- 
tained? Simply put—how do we learn and how do we re- 
member? Without memory, there can be no mind. 

How it is that the brain codes, stores and retrives mem- 
ories is among the most important and baffling questions in 
science. At the cellular level, there are two fundamental 
types of information coding. One of these is the familiar 
genetic code, shared by organisms from virus to man. In 
higher organisms, literally millions of bits of information 
are coded in the DNA of the cell nucleus. Over the course 
of evolution, a quite different kind of information coding 
has developed—the cellular encoding of acquired informa- 
tion in the brain. This coding is no less remarkable than the 
genetic code. It has been estimated that a well-educated 
adult human has millions of bits of acquired information 
stored in the brain. 

The generic term for acquired information coding in 
the brain is the “memory trace” or “engram.” The funda- 
mental difference between the genetic code and the engram 
code is, of course, that each individual human’s engram 
store is acquired through experience and learning. It is the 
biological substrate for the growth of knowledge and civi- 


lization. The individual uniqueness of each human being is 
due largely to the engram store—the biological residue of 
memory from a lifetime of experience. 

The development, elaboration and growth in size of the 
human brain over the past three million years is unprece- 
dented in evolution. Indeed, some biologists now feel that 
a new principle or mechanism of evolution may have come 
into play, a mechanism that involved the rapidly growing 
ability to learn, with its consequent development of culture 
and transmitted knowledge. The learning that occurred in 
each individual created a more complex environment that 
required still better learning, and hence a larger brain to 
survive and be fit. The cellular roots of this ability to learn 
can be traced to simpler organisms. It is best developed in 
mammals, where basic phenomena of learning, like Pavlo- 
vian and instrumental conditioning, exhibit the same funda- 
mental properties from rat and rabbit to human. The com- 
plexity of information processing and learning increases in 
an orderly fashion over the class of mammals. 

The ability to learn is an emergent property of cellular 
tissue. The ability, as such, has a clear genetic base; it is 
dependent on the structural and functional organization of 
the brain and on the elaboration of cellular storage pro- 
cesses. It would not be entirely surprising if the genetic 
material itself plays a role in the process of learning. 
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In the past generation, understanding of the biological 
basis of learning and memory has undergone a revolution. 
The earlier notion that memory is diffusely distributed in the 
brain has been largely discounted. It is now clear that vari- 
ous forms and aspects of learning and memory involve par- 
ticular systems and circuits in the brain. The realization 
means that it is now both theoretically and technically pos- 
sible to define these circuits, localize the sites of memory 
storage and analyze the cellular and molecular mechanisms 
of memory. 

The roots of this new understanding lie in several dif- 
ferent disciplines. From psychology has come a clear char- 
acterization of the behavioral properties of learning—what 
learning and memory are—and a developing theoretical 
analysis of the nature of the associative processes that form 
the basis of learning and memory. The work of such psy- 
chologists as Robert Rescorla at the University of Pennsyl- 
vania and Alan Wagner at Yale University provide exam- 
ples.'! The term “associative”, incidentally, describes the 
basic property of learning, the fact that stimulus or event 
becomes associated with another stimulus, event or re- 
sponse, as in learning a foreign language. 

From the broad field of psychobiology or behavioral 
neuroscience has come the recognition that identifiable neu- 
ral memory systems and circuits in the brain can be charac- 
terized and analyzed. One example of this is our own current 
work, to be described later, on the neuronal circuits that 
code basic associative learning in the mammalian brain. 
Another example is current work where the human disorder 
of amnesia is being modelled in the primate brain by such 
scientists as Mortimer Mishkin at the National Institute of 
Mental Health? and Larry Squire at the University of Cali- 
fornia at San Diego.° Still another example concerns how 
chemical and hormonal processes can modulate the storage 
and retrieval of memories, as shown in the work of such 
scientists as James McGaugh at the University of California 
at Irvine* and Stephen Zornetzer’ and Joel Davis® at the 
Office of Naval Research. From the field of Neurobiology, 
we are learning about the cellular, biophysical and molecu- 
lar mechanisms that underlie elementary forms of associa- 
tive learning in simple neural circuits in invertebrates in the 
work of such scientists as Eric Kandel at Columbia Univer- 
sity,’ and Daniel Alkon at Woods Hole.® 

The success of this collective approach has been the 
source of great optimism in the field; fundamental insights 
into the physical basis of memory will continue to be 
achieved over the next few years. Indeed, many of us feel 
that the field is now in the critical “breakthrough” stage— 
the present would seem to be the most exciting phase in the 
history of the field. 
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The Search For The Memory Trace 


The idea that memory traces can be localized to partic- 
ular circuits in the brain is not new. Karl Lashley began the 
search for the memory trace early in the century. He trained 
rats to learn mazes and then removed portions of the 
“highest” region of the brain, the cerebral cortex. But he 
was unable to find any one region of the cortex that seemed 
to be particularly involved in the memories for the maze 
habit. 

Because of his (and others) inability to find localized 
memory traces in the cerebral cortex, Lashley drew the 
following rather pessimistic conclusion at the end of his 
career: 

“This series of experiments has yielded a good bit of 

information about what and where the memory trace is 

not. It has discovered nothing directly of the real nature 

of the engram. I sometimes feel, in reviewing the evi- 

dence on the localization of the memory trace, that the 

necessary conclusion is that learning just is not possi- 

ble. It is difficult to conceive of a mechanism which can 

satisfy the conditions set for it. Nevertheless, in spite of 

such evidence against it, learning does sometimes 

occur 


With the advantage of hindsight it now appears that 
certain brain structures below the level of the cerebral cortex 
play a more important role in maze learning in the rat. One 
such structure is the hippocampus, a part of the so-called 
limbic forebrain system that also seems to be involved in 
amnesia in humans. 

As Lashley early recognized, the overriding problem 
for analysis of memory traces in the mammalian brain has 
been that of localization. In order to analyze neuronal/ 
synaptic mechanisms of memory storage and retrieval, it is 
first necessary to identify and localize the brain circuits, 
structures and regions that are critically involved. The gen- 
eral approach to localization of memory traces that has 
proved most productive is to use what are termed “model 
biological systems”—to utilize a form of basic associative 
learning in an animal preparation where the memory circuits 
can be identified and analyzed. 

Ivan Pavlov was perhaps the first to develop and use 
the model biological system approach to learning and mem- 
ory. From the time he discovered the conditioned reflex, he 
Saw it as a tool to investigate higher functions of the brain. 
Lashley, influenced by Pavlov, Bechterev and Watson, was 
the first Western scientist to state explicitly the model sys- 
tem approach, namely to use simply conditioned reflexes as 
models of associative learning and to localize memory 
traces by tracking the essential conditioned response cir- 
cuitry through the brain. 








Figure 1: 


Diagram of classical conditioning. Initially, the conditioned 
stimulus (CS) does not elicit the response to be condi- 
tioned (unconditioned response, UR). On the other hand, 
the unconditioned stimulus (US) or “reinforcement” does 
elicit the unconditioned response. If the conditioned stimu- 
lus (CS) is then paired with the unconditioned stimulus 
(US) such that the CS precedes the US by an appropriate 
time interval, then a predictive association is formed be- 
tween the two and the CS comes to elicit the conditioned 
response (CR). If the CS is then presented repeatedly 
without the US the CR is gradually forgotten or extin- 
guished. 
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Pavlovian or classical conditioning has come to be 
viewed as the basic process of associative learning. All 
readers are probably familiar with Pavlov’s discovery of the 
conditioned reflex—if a tone is sounded just before food is 


given to a dog, the dog learns to salivate to the tone—the 


tone signals or predicts the presentation of food. It is a basic 
way that all animals, including humans, learn about pre- 
dictability and cause and effect in the world. The tone is 
termed a conditioned stimulus (CS)— initially it does not 
elicit salivation. But when it has been paired with food, a 
biologically meaningful simulus that elicits a response di- 
rectly, the unconditioned salivation response (UR), then the 
tone comes to elicit salivation, the conditioned response 
(CR). The food is termed the unconditioned stimulus 
(US)—it directly elicits the salivary response. This is sche- 
matized in Figure 1. In this and many other examples of 
Pavlovian conditioning, the learned response of salivation 
(or eyeblink) to the tone CS is very similar to the unlearned 
response of salivation to the food US (or eyeblink to the 
corneal airpuff US). 

In the Western world, discrete behavioral responses 
like leg flexion or eyelid closure have been more widely 
used than autonomic responses like salivation to study the 
properties of associative learning. But they all show the 
same basic properties and phenomena of Pavlovian learning 
in all mammals, including humans, and in invertebrate 
preparations as well. The major breakthroughs that have 
occurred in the past few years in terms of identifying critical 
memory trace circuits and processes in both invertebrates 
and vertebrates have used Pavlovian conditioning proce- 
dures. These basic procedures, completely under the controi 
of the experimenter, are an important part of the model 
system approach. 

Some years ago, we adopted a particularly clear cut and 
robust form of associative learning in the intact mammal as 
a model system: Pavlovian conditioning of the rabbit eyelid 
response to an acoustic or visual CS using a corneal airpuff 
US. Thus, a tone CS is sounded and about a quarter of a 
second later a puff of air is delivered to the cornea of the 
eye. Initially, the tone does not evoke any response of the 
eye and the airpuff US evokes eyelid closure, a simple 
defensive reflex of the eye. Be repeating such pairings the 
tone comes to elicit a full blown closure of the eye, a 
conditioned response, that closely resembles the reflex eye- 
blink initially elicited by the airpuff. It is a highly adaptive 
learned response. Over a wide range of time intervals from 
tone onset to airpuff onset, e.g., from 1/10 second to more 
than a second, the animal (and human) learns to close the 
eye maximally at the time the airpuff to the cornea occurrs, 
thus providing maximal protection of the eye. 
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Classical conditioning of the eyelid response was first 
done in humans. Ernest Hilgard, now emeritus professor of 
psychology at Stanford, was the first to study eyelid condi- 
tioning in infrahuman animals in his classical work with 
Donald Marquis on dogs and monkeys. !° He and Marquis, 
then both at Yale, recognized immediately that eyelid condi- 
tioning provided an excellent model system for analysis of 
brain substrates of memory and they undertook a series of 
lesion studies with the assistance of John Fulton. They used 
a visual CS and showed that the visual cortex was not 
essential. Their results pointed to a subcortical memory 
trace. Isadore Gormezano, then at Indiana, was the first to 
publish eyelid conditioning studies in the rabbit, an animal 
that is docile and tolerates restraint well.'! The conditioned 
eyelid response in the rabbit has been used to very good 
effect in analysis of basic theoretical issues in learning. 

Classical conditioning of the rabbit eyelid response has 
a number of advantages for analysis of brain substrates of 
learning and memory, which have been detailed else- 
where.!? A particular advantage of the conditioned eyelid 
response is the fact that eyelid conditioning has become 
perhaps the most widely used paradigm for the study of 
basic properties of classical or Pavlovian conditioning of 
striated muscle responses in both humans and infrahuman 
subjects. It displays the same basic laws of learning in 
humans and other animals. Consequently, it seems highly 
likely that neuronal mechanisms found to underlie condi- 
tioning of the eyelid response in rabbits will hold for all 
mammals, including humans. We view the conditioned eye- 
lid response as an instance of the general class of discrete, 
adaptive behavioral responses learned to deal with aversive 
stimuli and adopt the working assumption that neuronal 
mechanisms underlying associative learning of the eyelid 
response will in fact be general for all such learning. 

In science, it is generally easier to rule out possibilities 
rather than rule them in, as Sherlock Holmes was fond of 
pointing out. We completed a long series of studies ruling 
out various brain structures and circuits as a part of the 
essential memory trace circuit or locus of the essential mem- 
ory trace itself. In brief, our results and work from other 
laboratories argued against essential participation of such 
higher brain structures as the cerebral cortex and hippocam- 
pus. Interestingly, memory traces do develop in the 
hippocampus as a result of such basic associative learning ,'4 
but this hippocampal memory trace circuit is not essential 
for learning. The development of such higher order memory 
systems in basic associative learning, incidentally, may pro- 
vide simplified models for the study of neuronal substrates 
of the more complex or cognitive functions of higher regions 
of the brain. 
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We also ruled out certain circuits in the brain stem as 
the locus of the memory trace, e.g., the primary auditory 
nuclei that relay information from the ear about the tone CS 
to other brain structures and the motor nuclei that generate 
the behavioral response, although portions of them are of 
course a part of the memory trace circuit. But we were still 
left with much of the brain stem, midbrain, and cerebellum 
as possible sites. Since there was no a priori way of deter- 
mining which of these regions and structures are involved in 
the memory trace, we undertook, beginning some years 
ago, to map these structures by systematically recording 
neuronal unit activity in already trained animals.'> For this 
purpose we developed a chronic micromanipulator system 
that permits recording of nerve cell activity in a substantial 
number of neural loci per animal. Increases in unit activity 
that form a temporal model within a trial of the learned 
behavioral response were prominent in certain highly local- 
ized regions of the cerebellum, in the cortex and in the 
interpositus nucleus. The results of the mapping studies 
pointed to substantial engagement of the cerebellar system 
in the generation of the conditioned response. An example 
is shown in Figure 2 with unit recordings from the cerebellar 
interpositus nucleus on the same side as the trained eye. This 
animal was given unpaired training before acquisition 
began. Average histograms reveal that the unit activity 
showed only minimal responses to the tone and airpuff dur- 
ing the unpaired day of training. However, during acq:‘isi- 
tion, as the animal learned, the unit activity developed a 
“model” of the conditioned response but no clear model of 
the unconditioned response. The cerebellar neuronal unit 
model of the learned response precedes the behavioral re- 
sponse significantly in time and predicts the actual 
amplitude-time course of the learned behavioral CR. The 
course of development of the conditioned behavioral re- 
sponse and the concomitant growth in the neuronal unit 
“model” of the conditioned response in the interpositus nu- 
clear region show very high correlations (e.g., r = .90).!° 

The neural unit recordings displayed in Figure 2 are 
from small groups or clusters of neurons, so-called multiple 
unit recording. In current work in which we record the 
action potentials from single neurons, we find two patterns 
of activity in neurons that respond in relation to the behav- 
ioral learned response: (1) those that show increases in fre- 
quency of discharge that precede and predict the form of the 
learned behavioral response, as in Figure 2, and (2) those 
that show mirror images of such responding (inhibition) but 
are equally predictive of the learned response. Neurons 
showing these positive or negative models of the learned 
response are found in very localized regions of the interposi-- 
tus nucleus and in very localized regions of the cerebellar 
cortex, where the neurons have been identified as Purkinje 
cells, the only neurons that send information out from the 
cerebellar cortex, in this case to the interpositus nucleus. 








Histograms of unit recordings obtained from a chronic 
electrode implanted at the lateral border of the interpositus 
nucleus. The animal was first given random, unpaired pre- 
sentations of the tone and airpuff (104 trials of each stimu- 
lus) and then trained with two days of paired training (117 
trials each day). Each histogram is an average over the 
entire day of training indicated. The upper trace repre- 
sents movement of the NM. The first vertical line repre- 


sents the onset of the tone CS, while the second line rep- 
resents the onset of the corneal airpuff US. Each 
histogram bar is 9 milliseconds in duration. Notice that 
these neurons develop a model of the conditioned but not 
unconditioned response during learning, and that this neu- 
ronal mode! precedes the learned behavioral response 
substantially in time (see reference 16). 
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Figure 3: 


Examples of cerebellar lesions that abolish the condi- 
tioned eyelid response: A, large lesion of both cortex and 
interpositus nucleus; B, smaller lesion of interpositus; C, 
lesion of the output pathway from the interpositus. Lesions 
limited to the same side of the cerebellum as the trained 
eye are sufficient to abolish the learned response (and 
prevent learning if made before training). Drawings are 
cross sections through the cerebellum and labels refer to 
various structures of the cerebellum and brain stem. 








In current work, we have found that lesions on the 
same side as the trained eye in the cerebellum (Figures 3 and 
4) permanently abolish the CR but have no effect on the UR 
and do not prevent subsequent learning by the other eye.'’ 
If the lesion is made before training, no learning occurs. The 
critical region is the lateral interpositus nucleus. Perhaps, 
most dramatic is a recent study involving kainic acid (it 
destroys nerve cell bodies but not nerve fibers or terminals); 
destruction of neuron cell bodies in a region as small as a 
cubic millimeter in the lateral portion of the interpositus 
causes complete and permanent loss of the learned eyelid 
response. '® 

Electrical stimulation through recording microelec- 
trodes in the critical lateral interpositus nuclear region of the 
cerebellum elicits a discrete eyelid closure response prior to 
training. In fact, a wide range of discrete behavioral re- 
sponses—eyelid closure, leg flexion, head turning—can be 
elicited by microstimulation of the interpositus, the type of 
response depending upon the exact location of the stimulat- 
ing electrode. 








Figure 4: 

Effects of ablation of left lateral cerebellum on the learned 
NMleyelid response (six animals). Solid triangles, ampli- 
tude of conditioned response (CR); open diamonds, am- 
plitude of unconditioned response (UCR). All training was 
to left eye (ipsilateral to lesion) except where labelled “R” 
(for right eye). The cerebellar lesion completely and per- 
manently abolished the CR of the ipsilateral eye but had 
no effect on the UCR. P; and Po indicate initial learning on 
the two days prior to the lesion. L1-L4 are four days of 
post-operative training to the left sg The right eye was 
then trained and learned rapidly (R), thus controlling for 
nonspecific lesion effects. The left eye was again trained 
and showed no learning. Numbers on abscissa indicate 
40 trial periods, except for “right eye” which are 24 trial 
periods (see reference 17). 





EFFECTIVE LESION 


i 
' 


LESION 


of 
st 
7 
6 
5 
4t 
3 
2 
1 
°o 


~ 
S 
a 
a 
= 
< 
x 
wi 
a 
= 
z 





P-2 P-1 PI P2 


TRAINING DAY & PERIOD 








Figure 5: 


The efferent (output) pathway from the cerebellar memory 
trace system to motor nuclei. 
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These results indicate that the essential memory trace 
circuits for these learned responses are extremely localized 
in the brain. How general is this. finding? Most of our work 
has been on the learned eyelid response, but we have shown 
that a different part of the interpositus is essential for 
hindlimb flexion conditioning.'? We and our associated lab- 
oratories are now exploring the generality of our results 
across learning procedures and species. Particularly impor- 
tant are current results in collaboration with Michael 
Patterson at Ohio University. He developed a procedure for 
eyelid conditioning in the cat, and has now found that the 
interpositus nucleus on the same side as the trained eye is 
essential for the learned response in this species, as well as 
in rabbit. It would seem that our results may be generaliz- 
able to all mammals. In other work, Patterson developed a 
procedure for “instrumental” avoidance learning of the eye- 
lid closure response in the rabbit, where eyelid closure to 
tone CS before onset of corneal airpuff results in the airpuff 
not being delivered. He found that lesions of the ipsilateral 
interpositus nucleus abolish this instrumental avoidance re- 
sponse, as well as the classically conditioned response. 
Hence, we feel we can generalize our essential cerebellar 
circuit to discrete, adaptive responses learned to deal with 
aversive stimuli in both classical and instrumental avoidance 
learning paradigms in mammals. 

In other work, we have identified much of the efferent 
or output pathway from interpositus nucleus to motor nuclei 
(Figure 5). Our results indicate that it courses out the supe- 
rior cerebellar peduncle, crosses to the contralateral side in 
the peduncle, relays in the magnocellular division of the red 
nucleus, crosses back to the ipsilateral side and projects to 
lower brain stem as a part of the descending rubral path- 
way.”° The essential CR circuit we have so far defined could 
be called the “efferent” limb in that destruction of any part 
of the circuit abolishes the CR to any CS (e.g., light or 
tone), but the association between CS and US/UR appears 
to be formed in the cerebellum, as will be seen below. 


A Hypothetical Model 

In late 1983, we developed a hypothetical schema or 
model of how the cerebellar memory trace circuit might 
work, based on the well-known anatomy of the system, on 
our results described above, and on theories of how the 
cerebellum might function as a learning machine (Fig- 
ure 6—see Figure 8 for a more realistic schematic of the 
cerebellar circuitry). The cerebellar cortex has great appeal 
to theoreticians and modellers because of its elegant unifor- 
mity and simplicity and because of the striking fact of two 
quite different inputs to each Purkinje cell—a mossy fiber- 
granule cell-parallel fiber input that is widely distributed and 
a climbing fiber input that is highly localized. All models, 
both verbal-logical and computational, have stressed this 
point, and those that focus on learning and memory have 
universally hypothesized that the mossy fiber-granule cell- 
parallel fiber system is the learning input, and the climbing 
fiber input is the “teaching” input.”! 








Scheme of hypothetical memory trace system for associa- 
tive learning of discrete, adaptive, somatic-motor re- 
sponses to deal with aversive unconditioned stimuli. Most 
interneurons are ommitted. It is assumed that the site of 
the memory trace is at the Purkinje cells shown in the 
upper left under “motor programs” and/or at associated in- 
terneurons. The locus of the memory trace is shown as 
cerebellar cortex and the basic notion is very similar to 
earlier theories of cerebellar plasticity (Albus, Marr, 
Eccles, Ito), but traces could also or alternatively be 
formed in the interpositus nucleus (int. n.), we assume by 
an a circuitry. A given CS (1 KHz) activates a 
subset of parallel fibers that in turn activate weakly all 
Purkinje cells shown. A different tone also activates all 
Purkinje cells but by a partially different group of parallel 
fibers. The US pathway is assumed to be via the inferior 
olive and climbing fibers. A given US is assumed to acti- 


vate only a limited group of Purkinje cells coding the 
motor program for the defensive response that is specific 
for the US (eyelid closure, leg flexion). When parallel fiber 
activation occurs at the appropriate time just prior to 
climbing fiber activation, the connections of the parallel 
fibers to the Purkinje cells activated by the particular US 
are strengthened or weakened. The efferent pathway from 
Purkinje cells to motor neurons is by way of the interposi- 
tus nucleus, superior cerebellar peduncle, and red nu- 
cleus. The scheme accounts for stimulus specificity, i.e., 
the fact that CRs shown a stimulus generalization gradient, 
for response specificity of learned responses, transfer ef- 
fects and is consistent with all evidence to date. Although 
hypothetical, insofar as being the substrate for memory 
traces, each aspect and assumption is amenable to ex- 
perimental test. 
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The details of our schema need not concern us here.” 
In brief, it is assumed that the locus of the memory trace is 
at the Purkinje cells in cerebellar cortex (and/or in analogous 
cells in the interpositus nucleus). A given CS (tone, light) is 
assumed to activate a subset of mossy fibers-granule cells- 
parallel fibers that in turn activate weakly a number of 
Purkinje cells. The US (corneal airpuff) is assumed to acti- 
vate a limited number of climbing fibers from the inferior 
olive that in turn activate strongly only a few of the Purkinje 
cells also activated by the CS, the few that result in activa- 
tion of the appropriate motor program via the interpositus 
nucleus, e.g., eyelid closure for a corneal airpuff or leg 
flexion for a paw shock. When parallel fiber activation 
occurs at the appropriate time just prior to climbing fiber 
activation, the connections of the parallel fibers to the 
Purkinje cells activated by the particular US are strength- 
ened. The schema accounts for stimulus specificity, e.g., 
the fact that CRs show a stimulus generalization gradient— 
if a different stimulus is used as a test CS the learned re- 
sponse will be weaker or nonexistent, the more different the 
test CS is from the training CS. It also accounts for response 
specificity of learned responses and is consistent with all our 
evidence to date. Although much of this circuit was hypo- 
thetical, insofar as it’s being a substrate for the formation of 
memories is concerned, each aspect and assumption is 
testable. Indeed, current work in our laboratory is providing 
strong new evidence favoring such a schema. 


The Inferior Olive-The Necessary and Sufficient 
Teaching Input for the Learning of Discrete Behav- 
ioral Responses 

A major system that projects information to the cere- 
bellum is the inferior olive (IO)-climbing fiber system (Fig- 
ures 6, 7 and 8). In recent work, we have found that lesions 
of the appropriate region of the IO (rostromedial DAO) do 
not abolish the CR but instead lead to relatively normal 
extinction with continued paired CS-US training.”* Lesions 
of all other regions of the IO do not affect the CR. The 
dorsal accessory olive (DAO) appears to be the essential 
afferent limb for the reinforcing or “teaching” input from the 
US. The fact that lesions of the DAO do not immediately 
abolish the CR but instead lead to its eventual extinction 
argues that the essential memory trace cannot be there. 

In current work, we find that electrical microstimula- 
tion of the DAO (see Figure 8) can elicit a variety of behav- 
ioral responses, including eyelid closure, the nature of the 
threshold response being determined by the exact location of 
the stimulating electrode. If this is now used as the US/UR 
and paired with a tone CS, the exact response elicited by 
DAO stimulation is learned to the tone as a CR rapidly and 
with all the properties of a normal CR.** Lesion of the 
criticai interpositus region abolishes this IO-established CR, 
and abolishes the response elicited by IO stimulation. Con- 
trol stimulation 1-2 mm dorsal to the DAO in the reticular 
formation can also elicit movements, presumably by activa- 
tion of descending reticular pathways, but these elicited 
movements cannot be trained to a CS. 
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Figure 7: 
Our hypothesis of the effective US reinforcing or teaching 
input circuit from the cornea of the eye to the cerebellum. 
The critical region of the inferior olive is the dorsal acces- 
sory Olive. 
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These IO results strengthen the argument that the IO 
and its climbing fiber input to the cerebellum is the essential 
US teaching input and that the trace is localized to the 
cerebellum. These are also the first clear empirical evidence 
supporting the pioneering hypothesis and network models of 
Albus, Eccles, Ito, Marr and others (noted in reference 21) 
that the IO-climbing fiber system is the “teaching” input for 
behavioral learning in the cerebellum. (Ito has developed 
analogous findings in the context of plasticity of the 
vestibulo-ocular reflex, and Llinds reports a similar role for 
the inferior olive-climbing fiber system in recovery from 
postural abnormalities following vestibular damage.*>) 


Creation of a Known CS Pathway—Mossy Fiber Pro- 
jections to Cerebellum 

We have succeeded in creating a known CS pathway by 
using electrical microstimulation of mossy fiber projections 
to the cerebellum as the CS (see Figure 8). Animals rapidly 
learn normal behavioral conditioned responses to this CS 
(e.g., eyeblink CR with corneal airpuff US). To date we 
have successfully used stimulation of mossy fibers from the 
dorsolateral pontine nucleus, the lateral reticular nucleus 
and the mossy fibers themselves, as CSs.7° 

Finally, in current pilot work we find that normal be- 
havioral CRs are learned with electrical stimulation of 
mossy fibers as the CS and of DAO-climbing fibers as the 
US, thus creating a “reduced” preparation within the intact, 
behaving animal (Figure 8). This preparation promises 
much in terms of fine-grained localization of the memory 
traces and analysis of mechanisms. 








Figure 8: 


Synaptic organization of the cerebellar cortex. Purkinje 
cells are excited directly by climbing fibers and indirectly 
(via parallel fibers from the granule cells) by the mossy 
fibers. Stellate and basket cells, which are excited by par- 
allel fibers, act as inhibitory interneurons. The Golgi cells 
act on the granule cells with feedback inhibition (when ex- 
cited by parallel fibers) and feedforward inhibition (when 
excited by climbing and mossy fiber collaterals). The out- 
put of the Purkinje cell is inhibitory upon the cells of the 
intracerebellar and vestibular nuclei. 

In normal behavioral training, a tone is used as the CS 
and a corneal airpuff, that elicits an eyeblink, is used a the 


US. However, electrical microstimulation of a portion of the 
inferior olive-climbing fiber projection to the cerebellum 
(see A in Figure 8) can also elicit an eyeblink response 
and serves as a very effective US in place of corneal air- 
puff. Similarly, electrical microstimulation of the mossy 
fiber projection to the cerebellum (see B in Figure 8) can 
serve as a very effective CS in place of tone or light. Fi- 
nally, joint stimulation of climbing fibers as the US and 
mossy fibers as the CS produces normal learning of dis- 
crete behavioral responses. See text for details. (Modified 
from Kandel, G.R. and Schwartz, J.H., Principles of Neuro- 
science, New York, Elsevier, 1984; Figure 30.3, p. 338.) 
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Conclusions 

In more general terms, our results have demonstrated 
quite clearly that the essential memory trace circuits in the 
brain for the basic category of associative learning we have 
studied are highly localized; and all our evidence to date 
points to discrete regions of the cerebellum as the locus of 
memory storage for discrete, adaptive behavioral responses 
learned associatively to deal with aversive events. 

The effects of cerebellar damage in humans are to 
impair movements, particularly skilled movements. Most 
movements that humans make are to a significant degree 
learned, i.e., skilled. In this context, Eccles has proposed 
the following: 


“We can say that normally our most complex muscle 
movements are carried out subconsciously and with 
consummate skill. The more subconscious you are in a 
golf stroke, the better it is, and the same with tennis, 
skiing, skating, or any other skill. In all these perform- 
ances we do not have any appreciation of the complex- 
ity of muscle contractions and joint movements. All 
that we are conscious of is a general directive given by 
what we may call our voluntary command system. It is 
my thesis that the cerebellum is concerned in this enor- 
mously complex organization and control of move- 
ment, and that throughout life, particularly in the earlier 
years, we are engaged in an incessant teaching program 
for the cerebellum. As a consequence, it can carry out 





all of these remarkable tasks that we set it to do in the 
whole repertoire of our skilled movements in games, in 
techniques, in musical performances, in speech, dance, 
song, and so on).”27 


One of the most striking features of the cerebellum is 
the high degree of regularity in the anatomical organization 
of the cortex over its extent and over species. It is highly and 
regularly organized; indeed, hard-wired. If a major function 
of this structure is to code the associative learning of dis- 
crete, adaptive movements and more generally skilled 
movements, then the basic neural circuitry for such learning 
preexists. No new pathways are formed. Instead, the mem- 
ory traces must involve changes in the excitability of preex- 
isting circuits. In this sense, at least, our results are closely 
consistent with studies of invertebrate preparations that ex- 
hibit learning, where the circuits are hardwired and the 
mechanisms appear to involve changes in membrane 
properties.78 

The general possibility that learning circuits are hard- 
wired in the mammalian brain is consistent with all we know 
about the high degree of anatomical organization of the 
brain and not inconsistent with all we know about learning 
and memory. Even the most complex form of human learn- 
ing and memory, language, appears to have differentiated 
anatomical substrates in the cerebral cortex and uniformities 
in the “deep structure” of language itself across all lan- 
guages.” In terms of mechanisms, the possibility that mem- 
ory circuits are hard-wired does not exclude anatomical 
substrates for memories in terms of the growth or alteration 
of the microstructure, i.e., change in the size and properties 
of synapses or even the development of new synapses. 

The work that we have reported here provides a partic- 
ularly clear example where theoretical computational mod- 
els derived from such fields as artificial intelligence and 
cognitive psychology can interact with and guide empirical 
neurobiological research. When we set out to find the essen- 
tial US teaching pathway, we had two major choices. Infor- 
mation about the corneal airpuff and other face skin sensa- 
tions is conducted into the brain by the Sth cranial nerve to 
the Sth nucleus. From here there are two major ways this 
information can get to the cerebellum: directly by way of 
mossy fibers and indirectly by way of the inferior olive and 
climbing fiber. The computational theories of cerebellar 
learning argued eloquently for the climbing fiber system as 
the teaching input because of its unique anatomical organi- 
zation, one climbing fiber to one Purkinje cell, as opposed 
to the distributed projections of the mossy fiber—granule 
cell—parallel fiber system to many Purkinje cells (see 
above). Following these models, we focused on the climb- 
ing fiber system and found that it was indeed the necessary 
and sufficient teaching input for the learning of discrete, 
adaptive behavioral responses. 

The distributed nature of the mossy fiber-granule cell- 
parallel fiber system, on the other hand, makes it ideally 
suited as the “learning” input, a point also stressed by the 
computational models. Again, following the models, we 
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have shown that activation of mossy fibers is indeed a very 
effective CS learning pathway. 

As we define the essential cerebellar memory trace 
circuit in more detail, it will permit us to develop more 
precise and powerful computational models that could well 
have more general and practical applicability, i.e., perhaps 
for control and guidance systems and for robotics. In terms 
of this last possibility, the manner in which our animals 
learn to make very precise and discrete movements to a 
“command” (tone CS) when electrical microstimulation of 
the inferior olive is used as the teaching US input is strik- 
ingly machine-like. As noted earlier, animals learn to make 
the exact movement evoked by the electrical stimulus and 
show no sign of emotion or affect when doing so. The 
cerebellum does indeed appear to be an extremely efficient 
biological machine for the learning of precise movements. 
Perhaps when we understand in sufficient detail how it ac- 
complishes this, we can make use of the same principles in 
non-biological machines. 

In general terms, it seems clear that as the neuronal 
circuits in the brain are defined that are essential for learning 
and other aspects of behavior, it will be necessary to develop 
theoretical computational models of these circuits. Even 
with very simple circuits, it is not possible to predict with 
any precision what in fact the circuit does or is capable of 
doing without quantitative modeling of the circuit.*° 
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Introduction 


The function of the brain is to process information 
which provides a basis for behavior. During the course of 
each day we have countless experiences. Our experiences 
create memories which constitute our representations of the 
world as well as our repertoires for behaving. Our knowl- 
edge is preserved in a variety of forms of memory. We learn 
and remember the unique events of the day, general facts, 
perceptual skills, and motor skills, as well as cognitive skills 
such as those required for problem solving, mathematics, 
and language. And, our ability to acquire new information 
rests heavily on our existing knowledge. Reading, under- 
standing, and remembering a sentence such as this, for 
example, involves the use of a highly complex set of well- 
learned skills. Clearly, the capacity to learn and remember 
is essential for information processing. 


Our research focuses on the neurobiological processes 
underlying learning and memory. Experiences have many 
effects on the brain. Some are fleeting and some are lasting. 
Most likely, only a portion of the effects will be involved in 
memory storage. Some of the effects no doubt serve to 
regulate or modulate memory storage. Understanding the 
mechanisms by which the brain stores information will re- 
quire knowledge of physiological systems which are acti- 
vated by experiences and which are causally related to cellu- 
lar changes that, we assume, underlie memory. The changes 
observed in nerve cells must also be causally linked to 
memory as it is expressed in behavior. Thus, the study of 
memory requires coordinated investigation at several levels 
of analysis—from cellular chemistry and physiology to be- 
havior. The study of memory also requires the development 
of a theoretical framework which will integrate the experi- 
mental findings. This paper summarizes our recent efforts to 
understand the neurobiological processes underlying the 
storage of experiences and to develop a conceptual model of 
learning which is faithful to the experimental findings. 
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Modulation of Memory Storage 


It is well-established that in humans as well as animals 
permanent memories are not established instantaneously at 
the time of an experience. The well-known clinical observa- 
tion that head injuries can produce retrograde amnesia indi- 
cates that memory storage processes remain susceptible to 
disruptive influences for some time following learning.' 
Experiments with laboratory animals have shown that mem- 
ory can be impaired by a wide variety of treatments affecting 
brain functioning, including electrical stimulation of the 
brain as well as drugs, if the treatments are administered 
shortly after training.? Typical retrograde gradients are 
shown in Figure 1. From one perspective, the susceptibility 
to retrograde amnesia would seem to suggest that the brain 
is badly designed. However, the evidence that memory can 
also be enhanced by posttraining treatments suggests that 
the susceptibility of the brain to posttraining modulating 
influences may have a highly adaptive function: Following 
an experience, memory storage processes may be modulated 
by endogenous physiological systems.* Our recent findings 
provide strong support for this view. Our evidence indicates 
that memory storage is modulated by hormones released by 
experiences and that the hormones act through influences 
involving brain systems that regulate memory storage. 





Figure 1: 


Gradients of retrograde amnesia and retrograde enhance- 
ment of memory. 
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Mildly stimulating conditions, such as those typically 
used in training animals, elicit the release of several stress- 
related hormones, including ACTH, corticosteriods, vaso- 
pressin, oxytocin, epinephrine, and norepinephrine, into the 
brain and blood. Several lines of evidence indicate that these 
hormones influence memory storage.° First, numerous stud- 
ies have shown that retention is influenced by posttraining 
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administration of these hormones. For example, in rats and 
mice, retention of recently learned information can be en- 
hanced by peripheral injections of low doses of epinephrine. 
High doses impair retention. Since such effects are seen 
only if the hormone is administered shortly after learning, it 
seems likely that the hormone affects retention by modulat- 
ing memory storage processes. Many experiments investi- 
gating hormone influences on memory have used a mild 
footshock as punishment in training the animals. For exam- 
ple, in a one-trial inhibitory avoidance task, animals are 
simply given a punishing footshock as they walk from one 
region of an apparatus to another. Retention is evidenced by 
the animals’ avoidance of the shocked area (measured as 
response latency) on a retention test. Posttraining 
epinephrine enhances the animals’ avoidance of the shocked 
area. Typical findings are shown in Figure 2. Comparable 
effects are obtained with other types of learning tasks using 
punishment, as well as tasks using a reward such as water 
for motivation. Thus, the effects of epinephrine on memory 
are not restricted to aversively motivated tasks. In other 
recent work, we have found that epinephrine enables ani- 
mals to learn under conditions where learning would other- 
wise not occur. Rats under general anesthesia were stimu- 
lated with legshock paired with a sound. Retention was 
tested later, following recovery from the anesthesia, by 
measuring suppression of drinking at the onset of the sound. 
Animals given epinephrine prior to training showed clear 
evidence of learning. The epinephrine did not alter the depth 
of anesthesia. Obviously, it will be of considerable interest 
to understand how and where epinephrine acts in enabling 


learning to occur in an anesthetized brain.® 


It is clear that posttraining hormones can affect mem- 
ory. But, do endogenously released hormones serve to mod- 
ulate memory? If they do, then learning and memory should 
be impaired by blocking the release of the hormones. 
Epinephrine is released from the adrenal medulla. We, as 
well as other investigators, have found that denervation of 
the adrenal medulla or adrenal demedullation impairs learn- 
ing of inhibitory and active avoidance tasks. Further, reten- 
tion in such surgically treated rats is normal if the animals 
are given posttraining injections of epinephrine.’ The find- 
ing that hormone replacement attenuates the retention im- 
pairment supports our interpretation that endogenous hor- 
mones are involved in modulating memory storage. 

There is extensive evidence that brain regions, located 
in the medial temporal region, including the amygdala and 
hippocampus, play an important role in memory storage. It 
is well-known that in monkeys as well as humans lesions of 
these brain regions produce a form of anterograde amne- 
sia—impairment in learning new information.*-? In rats, 
electrical stimulation of these brain regions produces retro- 
grade amnesia. '° In recent work we have found that adrenal 
denervation or demedullation alters the effects of amygdala 
stimulation on memory. Posttraining amygdala stimulation 
which produces amnesia in normal rats produces enhance- 
ment in adrenal-denervated or demedullated rats. Further, 
the effect is normalized, that is, the stimulation produces 





retrograde amnesia, if the animals are given epinephrine 
shortly before the amygdala stimulation.’ Such findings 
have suggested the possibility that endogenously released 
hormones may influence memory storage through effects 
involving the amygdala. 





Figure 2: 


Effects of posttraining administration of epinephrine of 
memory: dose response effect. 
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Other recent findings provide additional support for 
this view. For example, we have found that lesions of the 
stria terminalis, one of the major pathways connecting the 
amygdala to other regions of the brain, block the effects of 
peripheral epinephrine on memory.'! Since we have found 
that stria terminalis lesions also block the memory- 
modulating effects of the opioid antagonist naloxone, it may 
be that the effects of several hormonal systems are mediated 
by influences involving the amygdala. Other investigators 
have reported that, in rats, memory is modulated by intra- 
amygdala administration of opioid and adrenergic agonists 
and antagonists. In recent work, we have found that reten- 
tion is enhanced by injections of norepinephrine adminis- 
tered directly into the amygdala immediately after training 
(Figure 3).'? The effect is blocked by concurrent administra- 


tion of propranolol, a drug which blocks B-adrenergic re- 
ceptors. Such findings suggest that the effects of peripheral 
epinephrine on memory may involve adrenergic activation 
of the amygdala. In support of this view, we have found that 
posttraining intra-amygdala injections of propranolol block 
the effects of peripherally administered epinephrine on 
memory. Further, we have found that the memory- 
impairing effects of adrenal demedullation are attenuated by 
posttraining intra-amygdala injections of norepinephrine. 

Our findings are beginning to reveal the locus of neural 
systems through which hormones act in influencing memory 
storage. It seems highly likely that subsequent research will 
indicate that several brain systems are involved in modulat- 
ing memory. Further, the involvement of hormones in mem- 
ory is undoubtedly more complicated than is suggested by 
this brief summary of our findings. It is known, for exam- 
ple, that the effects of epinephrine on memory depend upon 
levels of other hormones, including corticosterone and vaso- 
pressin'>-!* as well as the opioid hormone B-endorphin (In- 
troini and McGaugh, unpublished findings). Such informa- 
tion should provide highly useful in efforts to understand the 
way or ways in which hormones modulate cellular activity 
at the sites in the brain where the structural changes under- 
lying memory occur. 





Figure 3: 


Effects of posttraining intra-amygdala administration of 
norepinephrine on memory: dose response effect. 
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Memory in Sensory Systems 

Two major informationally-based physiological disci- 
plines have emerged during the third quarter of this century: 
sensory physiology and the neurobiology of learning and 
memory. Unfortunately, they have trod different paths, with 
few intersections and a marked degree of mutual indiffer- 
ence. Happily, there are clear signs of a developing symbio- 
sis between these fundamental disciplines, some of which 
have emerged from our research program. It will be helpful 
to begin with a consideration of environmental events, stim- 
uli that are the inputs to the informational nervous system. 

Stimuli are described well by their physical parame- 
ters. In the case of acoustic stimuli, which we use, these 
include frequency (in KHz), intensity (in db referenced to a 
standard), location in space (referenced in azimuth and ele- 
vation to the intersection of the midline and interaural 
planes), and others which can be measured with equal preci- 
sion. However, physical parameters alone are insufficient to 
fully characterize stimuli that are the inputs to nervous sys- 
tems. Stimuli that impinge upon sensory receptors also have 
another set of parameters, their meaning or significance. 
These psychological parameters are measured by the behav- 
ioral output of nervous systems. Units of measure compara- 
ble to those used for physical parameters have not yet been 
developed for the psychological parameters of stimuli. 
Nonetheless, the standardization of many behavioral tech- 
niques has led to highly replicable findings among laborato- 
ries world-wide, and to an impressive amount of knowledge 
about the role of stimulus significance in neural information 
processing. For example, when an innocuous stimulus is 
given prior to a biologically important “reinforcing” stimu- 
lus, the first stimulus (“signal”) usually acquires the ability 
to elicit responses that previously were evoked by the sec- 
ond stimulus (“reinforcer”). This standard paradigm, termed 
Pavlovian conditioning in honor of its discoverer, has 
proven to be a powerful way of studying information pro- 
cessing during learning. 

In our studies of the effects of learning on the process- 
ing of sensory signals in the auditory system, we have made 
extensive use of such conditioning. It has proven highly 
advantageous to study the conditioned pupillary dilation 
response that develops when a sound is paired with cuta- 
neous stimulation. This response is an extremely sensitive 
and accurate index that the signal-reinforcer relationship has 
been learned (Figure 4).!>-!°-!7-!8.19 4 major advantage of 
pupillary conditioning is that the learned response develops 
within a very small number of pairings; even a single pairing 
is often sufficient to produce a learned association between 
the two stimuli (Figure 5)”° Pupillary conditioned responses 
belong to a class of acquired behaviors that all develop 
significantly faster than do specific muscular responses, 
such as those of eye blink and leg flexion.?!:22-3 In contrast 
to the latter class of specific responses, only one of which 
is acquired in a given learning situation, pupillary (and also 
cardiovascular, respiratory and non-directed muscular) con- 
ditioned responses are acquired regardless of the particular 
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stimulus that reinforces a signal. The brain processes which 
produce such rapidly acquired responses may well be neces- 
sary for the learning of eye blinks, leg flexions and the 
like.2!.24 





Figure 4: 


Pupillary dilation responses for a single cat as the interval 
between the beginning of an acoustic signal and the cuta- 
neous reinforcer is increased from 2.5 sec (A) to 16 sec 
(F). A rise in pupillometer output indicates increased dila- 
tion. Upward deflections indicate one second marker, 
downward vertical lines indicate auditory pips and cuta- 
neous stimulation. (The pupil response may appear to pre- 
cede the stimulus marker because of the curvilinear nature 
of the polygraph paper.) Note that the latency of maximal 
dilation increases as the interval increases, indicating that 
the animal has learned the relationship between the stimuli 
and so can predict the occurrence of the cutaneous stim- 
ulus. 
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Because of the rapidity and reliability of this learning, 
it has proven possible to record the electrical discharges of 
brain cells within the auditory system at the same time that 
an animal is actually acquiring a pupillary dilation condi- 
tioned response. As in the case of other sensory systems, the 
auditory system is comprised of neural stations that extend 
from its receptors at the periphery up through major subdivi- 
sions of the brain to the cerebral cortex. Our initial studies 
revealed that the responses of cortical neurons to acoustic 
signals changed systematically during learning.*> Further- 
more, a very high percentage of individual neurons exhibit 
this response plasticity (70-95%).7° The development of 
this neuronal plasticity was as rapid as that of the behavioral 
pupillary conditioned response. These findings indicated 
that the auditory cortex was responsive to the psychological 
significance of acoustic stimuli, not merely to their physical 
parameters. 





The finding of learning effects on the processing of 
acoustic information within the auditory system presented a 
paradox. It is universally agreed that sensory systems must 
provide an accurate and detailed representation of the envi- 
ionment. How, then, can a sensory system perform this 
function if its cellular responses to stimuli are governed not 
only by the physical characteristics of environmental stim- 
uli, but also by their acquired importance of signal signifi- 
cance? How, for example, is an animal able to distinguish 
between a loud unimportant sound and a weaker significant 
sound if its auditory neurons respond in the same way to 
both events? A resolution of this paradox was suggested by 
studies of the part of the auditory system which controls 
input to the auditory cortex. This is the medial geniculate 
nucleus (MGN) which comprises the thalamic auditory 
system. 





Pupillary learning curves. Data are divided into two groups 
of animals based on the rate of acquisition of conditioned 
responses. The slow learners were exposed to many 
acoustic stimuli prior to training; their acquisition of the 
pupillary dilation response is retarded because this prior 
exposure predicted the absence of a reinforcer. The rapid 
learners reveal the fundamental learning curve in the ab- 
sence of such prior inhibitory experience. Note that these 
subjects display conditioned responses within the first 10 
trials of the conditioning phase and reach asymptote by 
the 20th trial. The rate of acquisition is illustrated in finer 
detail for the first five trials of conditioning which shows 
that a conditioned response had actually developed on 
trial #2 relative to trial #1, hence one-trial learning. (Sensi- 
tization refers to a control period in which the acoustic sig- 
nal and the reinforcer were presented randomly). 
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Ascending auditory information is obliged to synapse 
in the medial geniculate nucleus on its way to the auditory 
cortex. Prior anatomical and physiological studies have re- 
vealed that this nucleus is actually comprised of subdivi- 
sions”’ that have different properties. Cells in the ventral 
part of the MGN are each narrowly tuned to specific acous- 
tic frequencies.”* Indeed, the frequency organization of the 
cochlea is mapped topographically in this region, as it is at 
all other levels of the auditory system, including the primary 
auditory cortex. In contrast, although cells in the medial part 
of the MGN also are responsive to auditory stimulation, they 
are not narrowly tuned. We recorded cellular responses to 
acoustic stimuli simultaneously in these subdivisions of the 
medial geniculate. Only neurons in the medial division de- 
veloped discharge plasticity; they did so rapidly, as quickly 
as the acquisition of the pupillary conditioned response. 
Cells in the ventral division were not plastic (Figures 6, 
7).79-30.23,31 Similar findings have been obtained in other 
laboratories using different species and different training 
tasks.**-33 So the paradox described above may be resolved 
by realization that the auditory system has anatomically and 
functionally distinct components which separately handle 
the physical and psychological aspects of stimuli. 





The medial geniculate nucleus showing its major subdivi- 
sions. Note that the ventral region (VL, VO) has small cells 
arranged in lamina. These neurons are each tuned to a 
narrow frequency band; they do not change responses 
during learning. The medial region (M) has “~~ cells, 
each of which responds to most frequencies. These cells 
are highly sensitive to the significance of sounds as indi- 
cated by rapid changes in response during learning. 














Figure 7: 

Histograms and pupillary records for a cell in the medial 
part of the MGN. Each histogram is the sum of neuronal 
discharges for two consecutive trials; each bar represents 
50 msec. Data are for the ten trials of sensitization and the 
first ten trials of conditioning. Note the decrement during 
sensitization and the extremely rapid increment during 
conditioning. The pupil response, as always, shows a de- 
crease during the former and an increase during the latter. 
Calibration: 6 spikes per division; horizontal bar, 1 sec. 
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These findings raise yet another problem. How are the 
physical and psychological aspects of auditory information 
processing brought together, so that ultimately meaning can 
be attributed to a particular stimulus, the details of that 
stimulus can be perceived, and both aspects of the stimulus 
can be stored? No definitive answer can yet be given, but it 
does appear that the two aspects of a stimulus are kept 
separate through the level of the thalamus. They appear to 
converge first at the auditory cortex. Thus, the auditory 
cortex is not strictly auditory, nor is it strictly cognitive. It 
seems to be both, and might better be termed “auditory 
perceptive cortex,” for perception is well-known to be influ- 
enced by both the physical and the psychological parameters 
of stimuli. 

Having established auditory cortex as a site of particu- 
lar interest, it became possible to combine the techniques of 
learning with those of sensory physiology to begin to deter- 
mine exactly which aspects of auditory information process- 
ing were altered by learning. Learning might simply change 
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the excitability of auditory cortical neurons. On the other 
hand, learning could alter information processing in a highly 
specific manner, perhaps by changing neuronal responses to 
particular acoustic frequencies only. Therefore, we initiated 
studies in which both techniques were employed and in 
which we asked whether learning causes specific changes in 
the frequency tuning of cells, for frequencies that we em- 
ployed as the conditioned signal during learning. 

The techniques of sensory physiology and learning are 


complementary. For the former, the physical parameters of 


a stimulus are systematically changed, while the meaning of 
a stimulus is kept constant. In the case of learning studies, 
the physical parameters are kept constant while the meaning 
is changed by arranging a relationship between the stimulus 
and a reinforcer. In our studies, we employ the sensory 
physiology approach before and after various phases of con- 
ditioning studies. First, for example, a neuron’s frequency 
response function is determined by presenting a series of 
tone bursts (at equal intensity at the tympanic membrane) 
from approximately 0.2 KHz to 25.0 KHz. (We have found 
that such frequency functions are very stable for cortical 
neurons in the absence of any behavioral training.) Next, we 
select one frequency to be used as the acoustic signal; this 
frequency is not the best frequency of the neuron, but one 
to which the neuron could in principle increase or decrease 
its discharges. We present this signal randomly with the 
cutaneous reinforcer. This training phase, referred to as 
sensitization, yields an estimate of the non-associative ef- 
fects of stimulus presentation. Following this test, we condi- 
tion the animal by pairing the signal frequency with the 
reinforcer. Pupillary conditioned responses develop at this 
time. Immediately thereafter, we again retest the cell with 
the full frequency range, as is done routinely in sensory 
physiology studies. We have found that conditioning can 
induce a frequency-specific change in the tuning function of 
a cell.*4 This change is generally highly specific to the 
frequency that was employed as the acoustic signal (Figure 
8). If we carry the training further, by presenting the acous- 
tic signal alone and deleting the reinforcer, the conditioned 
pupillary response disappears (“experimental extinction”). 
So also does the frequency-specific change in a cell’s tuning 
function. Thus, learning can involve a highly specific 
change in the processing of information about acoustic fre- 
quency. This change can be rapidly induced and rapidly 
reversed. 

These findings indicate that the effects of learning upon 
auditory cortex are not merely indicative of general changes 
in excitability, but actually engage mechanisms involved in 
the cell’s tuning. The data suggest that the receptive fields 
of cortical neurons may be more reflective of stimulus im- 
portance than of the physical parameters of the acoustic 
environment. Further, they reveal that a combined approach 
of learning and sensory physiology can be a powerful tool 
in the study of information processing in the brain. Studies 
of the mechanism of the specific effects of learning on 
information processing in the cortex need to be done. It is 
equally important to develop a comprehensive characteriza- 





tion of the learning effects, both in auditory cortex and in 
other sensory systems. Only then will we begin to develop 
adequate conceptualizations of the actual role of sensory 
systems in behavioral adaptation and the interplay between 
the physical and the physiological worlds. 





Learning produces a frequency-specific change in the 
“frequency tuning” function of single cells in the auditory 
cortex. For this neuron, the response to various frequen- 
cies (0.2-22.0 KHz) was determined at three times: before 
any training (“pre-sens”), following random presentation of 
twenty presentations each of a 6.0 KHz tone and cuta- 
neous stimulation (“post-sens”), and immediately after 
twenty paired presentations of these stimuli in the 6.0 KHz 
tone became a conditioned signal for the cutaneous stim- 
ulation (“post-cond”). (During pairing, the animal devel- 
oped a pupillary dilation conditioned response.) This par- 
ticular cell initially responded only weakly to some 
frequencies, and there was no systematic change follow- 
ing sensitization. In contrast, after learning that the 6 KHz 
tone was a signal, the cell greatly increased its response 
to this and adjacent frequencies. This indicates that the 
cell’s frequency response reflects the meaning of specific 
tonal frequencies. 
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Cellular Mechanisms of Memory 


Although sites of learning-induced response plasticity 
continue to be identified, we do not know what chemistries 
the brain uses to store memory. However, we can make 
some reasonable inferences about the site and general prop- 
erties of the mechanism. Most theorists conclude that the 
bulk of human memory is encoded as structural changes in 
discrete subpopulations of the thousands of billions of 
synapses in cerebral cortex (see reference 35, for a discus- 
sion). Structural changes are indicated by the duration of 
memory since the molecular constituents of the neurons and 
glia are broken down on a daily or weekly basis. Any purely 
biochemical change will therefore be replaced in a time 
frame far too short to account for memory. A synaptic locus 
for memory is required to account for the enormous capacity 


of memory. That is, if the whole neuron, or all of its axonal 
or dendritic processes were modified by single encoding 
events, the gradual accretion of memory over years should 
begin to deplete the pool of unchanged cells available for 
modification and hence storage. The same is true of 
synapses, but since these are present in 10,000 to 100,000 
greater numbers than neurons, the limit would be ap- 
proached much more slowly. Finally, cortex is chosen as the 
region of storage because it alone combines vast numbers of 
synapses, dense interconnectivity, and intimate access to all 
brain processing systems; the second and third of these 
features are required to deal with the ability of the memory 
system to associate disparate elements. Putting all this to- 
gether, we find that the memory chemistry should be capa- 
ble of irreversibly modifying the structure of cortical 
synapses and that it must be able to do this to a small group 
of contacts on a single cell without influencing the remain- 
ing inputs. This is a demanding set of constraints and one 
that can only be satisfied by a fairly exotic (to brain scien- 
tists at least) type of cell chemistry. 

Recently, members of our group have identified a 
specific enzymatic process that appears capable of produc- 
ing the effects described above, and have advanced the 
hypothesis that this process is responsible for certain types 
of memory. Before describing the hypothesis, it is necessary 
for us to first discuss a problem, in fact the problem that has 
dogged virtually every study of memory substrates in behav- 
ing animals. Consider this: It follows from the vast capacity 
of memory that any single item is encoded by changes in an 
infinitesimally small number of storage elements, but to 
detect biochemical changes we obviously need a very siz- 
able sample of changed elements. How then can we hope to 
observe the chemistry of memory in action? One approach 
that is enjoying increasingly widespread usage begins with 
the study of physiologically induced synaptic plasticity: The 
idea here is that if memory involves long-lasting synaptic 
modification elicited by neuronal impulses, then we should 
be able to produce the same effects (but on a massive scale) 
by stimulating brain circuitries with the appropriate patterns 
of activity. This strategy has been in the field for many 
years, but it was not until 1973 that investigators***’ ob- 
tained positive results. In those studies, it was shown that a 
few seconds of high frequency stimulation of the major 
input to the hippocampus, the simplest of cortical structures, 
produced an increase in the strength of the connections 
formed by that input and that this effect could persist for 
weeks and months. This was followed by an intense search 
for the types of synaptic changes associated with LTP, re- 
sulting in the discovery by one of the laboratories in our 
group, that the stimulation caused changes in both the shape 
of synapses and, most remarkably, the formation of new 
contacts.**-39 Other experiments have confirmed and ex- 
tended these observations.“°*! Alterations of these types 
can be expected to change the chemistry of synapses and, 
indeed, we now know that LTP is also correlated with an 
increase in the number of receptors for glutamate, one of the 
chemicals used as a transmitter in hippocampus.“ 
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How could intense synaptic activity produce effects of 
this type? This is a subject now under investigation in many 
laboratories, but one clue comes from the recent observation 
that injection in many laboratories, but one clue comes from 
the recent observation that injection into neurons of a chem- 
ical that buffers calcium to very low levels blocks the induc- 
tion of LTP.** This finding strongly suggests that the lasting 
potentiation is triggered by a sudden rise in intracellular 
calcium in post-synaptic processes. Summing up, it does 
appear that cortical synapses possess some type of chemical 
mechanism that is activated by physiological activity and 
that produces significant structural alterations. Moreover, 
we have reason to suspect that this mechanism is triggered 
by calcium and, among its other effects, produces an in- 
crease in the number of glutamate receptors. 


Membrane-Associated Calcium-Activated Proteases 

Most cell biological processes exist in a world of 
checks and balances and have effects that are fully re- 
versible. The proteases are a conspicuous exception to this 
rule; these enzymes break up their substrate proteins, an 
effect that can only be counteracted by replacing the protein. 
This irreversible effect of proteases makes them interesting 
candidates for long-lasting phenomena such as memory. 
Moreover, work over the past ten years has shown that many 
cell types, including neurons,“-*>- contain a group of 
calcium-activated proteases (“calpain”) that degrade 
proteins involved in forming the cytoskeleton,‘ an action 
that we can assume leads to structural change. Calpain acti- 
vation thus offers a means through which the types of alter- 
ations found in the LTP experiments might be realized, and 
there is now a considerable body of experimental work that 
supports this idea. 

First, the number of glutamate binding sites in synaptic 
membranes purified from hippocampus and cortex of rats is 
more than doubled when the membranes are exposed to low 
levels of calcium.**-*? This effect is irreversible in that the 
added receptors remain after the added calcium is re- 
moved.~°-*8 Thus, calcium, the suspected trigger of LTP, 
reproduces one of the correlates of LTP; moreover, cal- 
cium’s effect on receptors is selectively and dramatically 
blocked by inhibitors of calpain.*!-*? Biochemical experi- 
ments have demonstrated that calpain is present in high 
concentrations in synaptic membrane fractions,°* where it 
acts to degrade microtubule associated proteins (MAP’s) 
and fodrin (or brain spectrin).°* Fodrin, like all members of 
the spectrin family, forms a dense meshwork immediately 
beneath the membrane and serves to link the cytoskeleton 
with transmembrane proteins. There is considerable evi- 
dence that this meshwork restricts the mobility of membrane 
elements such as receptors. Proteolytic action in the fodrin 
network would be expected to produce changes in surface 
chemistry and thus could explain the effects of calcium on 
glutamate receptors. To directly test this idea, Siman et al.™ 
used antibodies against fodrin to protect it from calpain and 
found that under these conditions calcium no longer induced 
glutamate binding sites. 
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Calpain, by locally disrupting the cytoskeleton, might 
also produce structural reorganization of synapses as well as 
prepare membranes for the development of new contacts. It 
is difficult to test this idea in brain but recent experiments 
strongly suggest that shape change and structural reorgani- 
zation in red blood cells* and platelets are accompanied by 
calpain-mediated breakdown of spectrin and other 
cytoskeletal-linking proteins. 

These observations suggest the following hypothesis 
about the cell biology responsible for producing lasting al- 
terations in cortical synapses: 

(1) high frequency stimulation transiently increases in- 

tracellular calcium, thereby activating calpain; 

(2) calpain degrades the links between membrane and 

cytoskeleton; 

(3) these effects result in the appearance of new recep- 

tors and anatomical changes in synaptic contacts; 

(4) calcium homeostasis is rapidly restored but the ef- 

fects of calpain activation remain. 


Calpain and Memory in Cortical Networks 

There is much in the proposed calpain hypothesis as 
described immediately above that requires futher testing. 
We need to know much more about the precise localization 
of the enzyme and its substrates—it will also be necessary 
to determine if it is activated by physiological events and 
produces lasting changes in synaptic physiology. But the 
experimental results to date indicate that calpain provides an 
excellent means for selectively and irreversibly changing 
synapses and thereby offers a very specific candidate for a 
memory mechanism. It is therefore only appropriate to 
begin investigating the possible role of the enzyme in mem- 
ory. Unfortunately (for the experimenter) all behavioral 
paradigms may not be equally appropriate in this regard. 
Many investigators now believe that humans have at least 
two fundamentally different memory systems, one which is 
involved with specific data (e.g., names, faces, telephone 
numbers, locations, etc.) and a second which is concerned 
with the procedures or rules needed to solve a problem (e.g., 
operating a car, solving puzzles of the same type, etc.). 
Note also that it is possible that still other categories exist 
and that some of the most typically used paradigms may 
sample memories that are neither “data” nor “procedural” in 
type. This is particularly relevant here because it is very 
unlikely that calpain is involved in the production of all 
variants of information storage: We cannot detect the en- 
zyme in fish brains*’, and fish certainly learn. Even within 
mammals, calpain varies greatly across brain regions and 
cell types, as shown by immunocytochemical experi- 
ments,** and it does not appear to expose glutamate recep- 
tors in certain brain stem regions.*! Thus, any forms of 
memory stored in the lower parts of the brain may also 
involve processes other than calpain. Accordingly, we de- 
cided to begin our experiments using memory problems that 
clearly involved circuitries running through mammalian 
forebrain and in particular cortex, areas in which we are 
confident that the calpain mechanism is present. This led us 





to the olfactory system, since its projections have been well- 
studied and are restricted to forebrain regions. In our first 
studies, we injected an inhibitor of calpain, leupeptin, into 
the cerebral ventricles using continuously active osmotic 
minipumps implanted under the skin of rats. The drug 
caused no observable effects on feeding, drinking, body 
temperature, or exploratory behavior, but it produced a sub- 
stantial impairment of the rat’s ability to remember which of 
two odors led to a water reward.*?™ To test the generality 
of this result we measured spatial memory using an 8-arm 
radial maze; while little is known of the circuitries used by 
rats to solve this problem, it is well-established that the 
hippocampus is critical.*' Leupeptin produced a sizable im- 
pairment in the rat’s performance in this test.*? In striking 
contrast to these results, the calpain inhibitor had virtually 
no effect on shock avoidance learning. This dissociation 
raised again the possibility that different forms of memory 
with different chemical substrates co-exist in the rat brain. 
In support of this, we have found that drugs that do block 
shock-avoidance memory have no effect on olfactory mem- 
ory®. This provided the first double dissociation of memory 
using pharmacological agents and confirms the selective 
nature of the effects produced by leupeptin. 

Calpain pharmacology is very limited at this time al- 
though new drugs are gradually being developed. Future 
studies of the role of the enzyme in memory will be greatly 
facilitated when these become available for testing. But 
beyond this, we need to begin testing for the effects of drugs 
on the physiology of those circuitries involved in the encod- 
ing and storage processes. This will allow us to determine 
if calpain inhibitors actually do interfere selectively with 
behaviorally induced synaptic plasticity; moreover, if the 
projections and connections underlying the memory process 
can be identified, it may be possible to look for structural 
and chemical events, including calpain activation, associ- 
ated with storage. As discussed in the next section, the 
olfactory system is uniquely suited for analyses of this type 
and has a number of other advantages as well. 


Olfaction and Memory in Cortical Networks 

The projections of the olfactory system are relatively 
simple. Fibers from the nasal receptors terminate in the 
olfactory bulbs which in turn distribute their axons to the 
amygdala and an extended sheet of cells that constitute the 
outermost layer of the piriform/entorhinal (olfactory) cor- 
tex. This latter region then innervates the hippocampus and 
the dorso-medial nucleus of the thalamus. The smell system 
is unique among the senses in having virtually direct con- 
nections with three structures (DMN, amygdala, and 
hippocampus) involved in the processing of certain kinds of 
memory in mammals, including humans. This may explain 
why odors are so potent in evoking memories and may also 
explain the recent findings that olfactory problems are com- 
mon in human amnestics.°*™ It is also interesting that smell 
memory in rodents displays a number of striking similarities 
to everyday memory in humans. Recently we demonstrated 
that rats with lesions separating olfactory inputs from the 


hippocampus exhibit a rapid forgetting syndrome that paral- 
lels the anterograde amnesia reported for humans with dam- 
age to the temporal lobes and hippocampus. These animals 
were able to acquire smell discriminations but within an 
hour gave no evidence of remembering the correct odor. 
Work from other laboratories strongly suggests that this type 
of lesion does not interfere with the animals’ ability to learn, 
remember, and use procedures needed to solve olfactory 
problems. Thus, damage to the hippocampal system is se- 
lective to one kind of memory in rats as it is in humans (see 
reference 65 for a review of the human literature). 

The reason that olfaction is not widely used in learning 
experiments is that odors cannot be used as punctate, well- 
defined (temporally or spatially) cues; this vastly compli- 
cates the job of measuring physiological responses in the 
brain. However, preliminary studies suggest that it may be 
possible to substitute electrical stimulation of the olfactory 
bulb for odors as cues in rats trained on a series of olfactory 
discriminations (unpublished observations). If this proves 
correct, then it will be possible to exploit the extreme sim- 
plicity of the olfactory system to follow the events involved 
in processing and storing information in a cortical network 
and to test where and how drugs influence these events. 
Together with new pharmacological tools, those develop- 
ments should permit much more rigorous correlational and 
manipulative tests of the calpain hypothesis. 


Computational Psychobiology 

The field of the neurobiology of learning and memory 
has two sides: one is the question of neurobiological mech- 
anisms underlying animals’ information-processing abilities 
(i.e. , the ability to acquire, store and retrieve memories); the 
other, reciprocal question is: precisely what are these 
information-processing abilities themselves? In a simple 
sense, animals can store and retrieve information based on 
experience. But there are notable constraints on those abili- 
ties; not all experiences are learned with equal facility, and 
not all things that are learned are recalled accurately. Work 
in our laboratory on computational psychobiology has at- 
temoted to identify formal characterizations of precisely 
what is learned, and under what circumstances, and to relate 
those computational findings to neurobiological mecha- 
nisms that may underlie them. 

For example, a rat in a discrimination-learning envi- 
ronment learns over trials to pick out the key sensory feature 
cues that accurately predict a shock, or a food reward, etc., 
from among a wide array of possible candidate cues. At the 
neurobiological level, we might want to ask how it is that 
this discrimination ability arises from brain function. But at 
the computational level, we also must ask which of the 
many candidate sensory cues are learned, and why? There 
are many reasonable algorithms for a human or animal to 
use in selecting some subset of a set of candidate cues, and 
each different algorithm makes different predictions about 
what the animal will actually learn from particular experi- 
ences. Before we can attempt to find a circuit that perforn.s 
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this learning behavior, it would help to know precisely what 
it is that the circuit is supposed to compute, i.e., to be able 
to specify formally what the learning behavior actually is. 

There are a number of model systems of human asso- 
ciative learning behavior, ranging from sea slugs, to rats, to 
artificial-intelligence computer models. For any such model 
to be a useful substrate for the study of human cognition, it 
must contain similarities to human processing; this is indeed 
the motivating assumption in all model work. It is hoped 
that sea slugs have neural circuitry that works according to 
the same mechanisms and principles as human neural cir- 
cuitry; the same applies to pigeons, rats, and rabbits. There 
is, of course, no such hope for computers, yet it is possible 
to study human cognition on at least three levels via com- 
puter models: (1) At the biochemical level, computer mod- 
els can aid in the analysis of low-level mechanisms of 
synaptic change; (2) At the circuit level, models of connec- 
tivity and information-flow can aid in the testing of theories 
of neural networks; (3) At the behavioral level, accurate 
characterization of the constraints on what is learned can aid 
in the testing of theories of associative conditioning. Each 
level, in isolation, has shortcomings; accurate behavioral 
characterization can make only a limited contribution to- 
ward the development of a theory of the mechanisms of 
learning, and a biophysical mechanism of synaptic change 
will answer only a part of the larger question of how expe- 
riences are encoded, indexed and retrieved. 


What is Computed In Contingency—Based Associative 
ino? 


Our work has focused primarily on the level of behav- 
ioral characterization of learning. Our recent work has at- 
tempted to formalize results on contingency and latency in 
classical and instrumental conditioning. At the computa- 
tional level, a good deal of work on animal learning and 
memory™-®7-°8. has identified necessary constraints on be- 
havior, i.e., input-output specifications that must be ac- 
counted for by any algorithms intended as theories of contin- 
gent learning and memory. The constraints focused on in 
this section are those of the salience assignment task, i.e., 
identifying which of several features are predictive of a 
particular result. 

Imagine a rat attempting to decide, over trials, which 
of several environmental features or events should be 
learned to be a predictor of a recurring shock event. We can 
view the animal’s task as having to hypothesize potential 
prediction relationships between the shock and the various 
features, individually and in various combinations, and to 
weigh these hypothesized relations against each other. 

We might initially assume that a particular feature or 
event would be inferred to be the predictor of the shock 
depending on the number of times that feature actually oc- 
curred immediately before the shock. Each time the feature 
is paired with the shock, the association between the feature 
and the shock might be strengthened (see reference 70). 
Extensive experimental evidence in the psychological litera- 
ture shows this to be false. This condition is insufficient to 
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warrant the animals’ inference that the tone is more likely to 
predict the occurrence of shock. Rather, this association 
depends upon the fact that, of all stimuli present, the tone is 
the best predictor of the shock. Hence, the naive idea that 


‘the number of pairings alone determines the predictiveness 


(or salience) of candidate predictive features, or that 
strengthening alone could be the mechanism for learning 
associations, is false. 


Contingency vs. Number of Pairings 

This result requires a somewhat counterintuitive com- 
putation on the part of the animal: not only must the animal 
be computing, somehow, the conditional probabilities of the 
shock occurring, given the light and given the tone, but it 
must also be able to compute the relative conditional proba- 
bilities (RCP), i.e., the comparison or ratio of the likelihood 
of various stimuli preceding the shock. 

Saying that this “must be computed” speaks only of a 
behavioral constraint, but does not yet address the question 
of the algorithm that the animal might be using to arrive 
eventually at behavior that conforms to the constraint. 
Marr’s three levels of explanation for theories of visual 
processing’! can be extended to the domain of learning and 
memory: At the computational level, we may speak simply 
of the computations that must somehow be arrived at (such 
as the contingency or salience relationship between cues and 
results), without reference to how those computations may 
be carried out. At this level, we simply specify accurate 
empirical generalizations that capture the data. The al- 
gorithm level constitutes the level of mathematical function 
that performs the necessary computations. Finally, these 
algorithms are instantiated in hardware (neural or computer) 
at the implementation level. 

These three levels are not wholly independent. In par- 
ticular, the algorithm level must conform to the constraints 
provided by each of the other two levels: it must compute all 
and only those things that have been identified as actually 
occurring in humans and animals (at the computational 
level), and the proposed algorithms must be able to be 
carried out somehow, given the tools and limitations pro- 
vided in the substrate (at the implementation level). Res- 
corla’s’*:”3:4 formulation of the necessary computation un- 
derlying contingency is that the probability of the shock 
outcome (the US), given the conditional stimulus feature 
(the CS, e.g., the tone), must be greater than the probability 
of the outcome occurring without that feature having oc- 
curred, or p(US|CS) > p(US|CS). 

It is this measurement of relative conditional probabil- 
ities that accurately captures the empirical results of studies 
on contingency-based learning. Animals, and humans in 
analogous circumstances, exhibit contingency-driven learn- 
ing in the sense that they somehow maintain incrementally- 
updated knowledge of the relative predictiveness or salience 
of features. Through experience the animal must pick out 
the relevant (salient) features from the background of uncor- 
related features and use only these salient features to predict 
future events. 





Thus, although the term ‘contingency’ is often used 
simply to denote the selection (by any algorithm) of salient 
features from a group of candidate features during associa- 
tive learning, true contingency-based learning can be more 
formally stated as the assignment of associative salience to 
a feature (or feature-cluster) El relative to an outcome E2 
always and only when the conditional probability of E2 
occurring given the occurrence of El is greater than the 
conditional probability of E2 given the absence of E1, or as 
it was stated above, p(E2|E1) > p(E2\E1). 

Differential Predictions 

Because we have stated the constraint of true contingency 
in a formal way at the computational level, we can differen- 
tiate between algorithms that conform to this formal con- 
straint (i.e., conform to this accurate characterization of the 
data) and those that do not. Some particular algorithm might 
loosely account for associative learning phenomena by 
providing a mechanism for selecting salient from irrelevant 
cues, but that algorithm might not accurately conform to the 
actual data on contingency, as captured in the statement of 
relative conditional probabilities. 

In particular, algorithms that operate by strengthening 
associations by some amount each time a specific cue is 
paired with some outcome, and weakening them at non- 
pairings, cannot in principle compute true contingency. This 
is primarily because there is no differential weighting per- 
formed for different types of non-pairings, i.e., extra spuri- 
ous CSs versus extra spurious USs over trials; in general, 
linear equations cannot correctly compute contingency, 
which is a nonlinear equation. 

Granger, Schlimmer and Young” give a detailed ac- 
count of our own formulation of the problem and algorithm 
for the correct computation of contingency using only 
“local” calculations, that is, not making any assumptions 
about any extraordinary ability of the animal to hold lots of 
data in its head. To summarize the results, there are four 
categories of possible relationship among stimuli that are 
calculated: for any given US, a CS may be either a positive 
cue (i.e., predicts the US), a negative cue (predicts the 
absence of the US, i.e., a ‘safety signal’), a context cue 
(behaves in an ambiguous way with respect to prediction), 
or an uncorrelated cue (i.e., is random with respect to the 
US). 

We have constructed a working computer model of this 
algorithm, which we call CAP-CEL (Contingent Associa- 
tive Processing in the CEL framework).’° Figures 9 and 10 
illustrate the different predictions that would be made by 
(a) a true contingency algorithm (in this case our CAP-CEL 
algorithm) versus (b) a strengthening (linear) algorithm. The 
white and black items represent two different candidate cues 
that are sometimes paired with a shock; the animal is trying 
to decide which of them is the predictive cue. Animals 
would choose the white cue as the predictive one over the 
black cue, since the white cue is the one that is paired 
contingently with the shock. The stars on the graph in Fig- 
ure 9 illustrate that a computer simulation of CAP-CEL’s 


algorithm based on relative conditional probabilities cor- 
rectly orders the two cues, white over black. The circles in 
Figure 10, however, denote a computer simulation of a 
strengthening algorithm (one that strengthens or weakens 
the associative strength of cues based on the number of 
times the cues have been paired with the shock via a linear 
equation). This algorithm chooses the black cue over the 
white cue for this example set of trials. Hence, this associa- 
tive learning algorithm does not correctly model the animal 
data; i.e., it does not compute the right thing. Our 
computational-level approach to learning and memory 
therefore provides a set of analytical tests against which 
proposed mechanisms of learning can be measured. We are 
currently attempting to analyze a number of proposed ani- 
mal models of learning, including Hawkins and Kandel’s 
Aplysia circuits’, Alkon’s Hermissenda circuits”, and 
Lynch’s mammalian hippocampal slice circuits™*?, as well 
as a number of proposed computer and analytical models, 
including Gluck and Thompson”, Sutton and Barto®, and 
Gibbon et al.*', to identify precisely what these models 
compute. 

Thus, algorithmic interpretations of empirical results in 
neurobiology must conform both to “top-down” constraints 
of behavioral data (i.e., they must compute the right thing) 
and to “bottom-up” constraints of the substrate (e.g., known 
constraints of various brain structures and circuits). 





Figure 9: 

Discrimination learning curve using contingency. The open 
symbols denote acquisition of association to the predictive 
cue; closed symbols, to the unpredictive cue. Associative 
strengths greater than one indicate predictiveness. Note 
that the contingency mechanism correctly learns the pre- 
Cictive cue. 
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All experimental research in neurobiology is (at least im- 
plicitly) guided by some top-down hypotheses. At their sim- 
plest, these hypotheses can simply be statements such as: 
There are long-lasting changes in behavior during learning 
(at the computational level), so there must be some mecha- 
nism for causing long-lasting changes in the brain (al- 
gorithm level) which is grounded in some biophysical pro- 
cess (implementation level). However, any algorithm or 
neurobiological (implementation) mechanism proposed for 
long-lasting changes must conform to more stringent com- 
putational requirements than that the change simply lasts a 
long time once it has occurred. Computational-level data on 
the duration of the trace, its onset, and its extinction are 
crucial (and measurable) constraints on any such candidate 
mechanism; further crucial constraints are that the trace 
must be part of a system that enables the trace to be 
“retrievable” exactly when appropriate at some later time, 
the ability to store “sequences” of long-lasting traces, and a 
mechanism for “generalizations” of traces. These computa- 
tional constraints on a memory mechanism are more diffi- 
cult to collect data on, and yet any proposed mechanism that 
does not conform to them is clearly not the mechanism of 
long-term memory. It is crucially important for the compu- 
tational work formally specifying these constraints to move 
forward, so that candidate mechanisms for long-term mem- 
ory may be evaluated by these formal specifications. 





Figure 10: 


Discrimination learning curve using strengthening. The 
open symbols denote acquisition of association to the pre- 
dictive cue, and closed symbols denote acquisition of as- 
sociation to the unpredictive cue as in Figure 9. However, 
note that the strengthening mechanism incorrectly learns 
the unpredictive cue. 
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Concluding Comments 

Rapid progress in the subfields of memory research is 
beginning to afford opportunities for the development of 
synthetic theories in which various levels of analysis (and 
phenomenological observation) inform and interact with 
each other. Thus, we now have good evidence that certain 
cortical regions are involved in storage and that the 
hippocampus and amygdala promote the encoding process. 
How do these latter two structures influence physiological 
events in cortex? In what way is the transmission of infor- 
mation passing through olfactory and auditory cortices mod- 
ified by hormonal states acting upon the amygdala? It should 
be possible to begin addressing questions about the interac- 
tions at the level of neuronal systems. Similarly, tools are 
available for analyzing the similarities and differences be- 
tween memory types found at the behavioral level. We have 
found drugs that dissociate some categories of memory 
tasks. This can be carried much further, particularly if we 
add hormonal treatments to the list of manipulations. Stud- 
ies at a still more conceptual level can also be envisioned. 
As we more clearly define the various components of the 
neural systems that cooperate in producing memory, it will 
become possible to match these against the subcomponents 
of models of memory processing. It is also the case that the 
neural networks involved are in some cases so precisely 
defined that they can be used to construct computer models 
of memory machines. 
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Historically, in America neuropsychology and cogni- 
tive psychology have been distinct enterprises, with neu- 
ropsychology being largely clinical in focus and cognitive 
psychology being experimental in focus. However, cogni- 
tive methodologies have been profitably used in conjunction 
with clinical populations in Europe for decades.':? The 
value of combining the two approaches is now becoming 
apparent to a wide audience, as witnessed by the founding 
last year of the journal Cognitive Neuropsychology. How- 
ever, the power of cognitive/neuropsychological method- 
ologies is greatly amplified when combined with computa- 
tional theorizing. Computational theories specify the 
components of information processing in a way that can be 
programmed into a computer, and hence are precise and 
generate explicit predictions. There has been dramatic re- 
cent progress in studying cognitive processing due to this 
new approach. In this paper, we will illustrate this 
“computational neuropsychological” approach by consider- 
ing one cognitive activity, mental imagery. 

To look at them, computers have almost nothing in 
common with brains, and it is not clear how neuropsychol- 
ogy or cognitive methodologies can be augmented by con- 
sidering ways of programming computers. To see the salu- 
briousness of a combined approach, we must first appreciate 
the relevance of computers to understanding the brain. It 
may be most useful to begin by considering the stomach. 
Would you be satisfied that you had truly understood the 
nature of the stomach if all you had was a detailed descrip- 
tion of its physical composition? Say you knew its precise 
shape, which cells excreted which enzymes in what 
amounts, and so on; would that be enough? Probably not. 
You probably would want to know what all of this was for. 
Understanding why the stomach is constructed as it is re- 
quires an understanding of its role in digestion. You need to 
know what kinds of material go into the stomach and what 
are the requirements of organs that receive the output from 
the stomach. To truly understand the stomach, you need to 
understand the “problems” that it must “solve” (breaking 
down specific compounds into specific constituents) in the 
context of the digestive system as a whole. 

Similarly, a complete understanding of the brain will 
require more than a description of its physical composition 
(cells and their connections, various chemical and electrical 
interactions); if someone provided you with only a complete 
physical description of the brain, including a detailed 
blueprint of all the connections and pathways, this probably 
would not be entirely satisfying. We also need to know what 
all of this is for. The function of brains (or at least cortices, 
which “mental” functions carry out) is to process informa- 
tion. That is, the brain is the organ that allows us to recog- 
nize friends and foes, to store information, to communicate, 
and so on. The way it carries out these tasks involves the 
processing of incoming sensory information and the use of 
stored information. Thus, a complete understanding of the 
brain requires an understanding of how it processes informa- 
tion. 


The notion of function is, of course, hardly foreign to 
neuroscientists. But the language of computation provides 
the only precise and rigorous vocabulary for understanding 
information processing. By studying what seems to be re- 
quired to build a computer model, we learn something about 
the nature of organic intelligences. Constructing computer 
programs that process information the way brains do helps 
us to discover the problems that must be solved for an entity 
to see, learn, and so on, which are the analogues to the 
stomach’s job of breaking down foods into usable con- 
stituents. 

Thus, although they may not look alike, at one level of 
analysis computers work like brains. But a computer is a 
general purpose machine; it operates only in accordance 
with precise instructions. A computer will mimic a brain’s 
information processing only if one programs (instructs) it 
appropriately. How does one develop a theory that can guide 
construction of a computer simulation model? A promising 
answer to this question was provided by the late David 
Marr.’ In this paper, we will consider how Marr’s approach 
can be applied to cognitive processes, using mental imagery 
as a concrete example. Thus, let us now consider the bene- 
fits of a computational neuropsychology of mental imagery. 


Mental Imagery 

Visual mental imagery is a kind of “seeing” in the 
absence of the appropriate immediate sensory input; it is 
“re-seeing” information previously seen and stored in mem- 
ory. Imagery is an important component of many kinds of 
thinking. For example, one can anticipate the trajectory of 
a moving object by imaginally projecting its path, or one can 
discover an etficient way of packing irregularly-shaped ob- 
jects into a container by imagining them arranged in differ- 
ent ways. It is important to understand imagery not only for 
practical reasons (e.g., training people to be better naviga- 
tors), but because it is a fundamental cognitive ability that 
is used in numerous tasks. 

As the vocabularly used to characterize imagery sug- 
gests, there are some deep similarities between imagery and 
like-modality perception. Consider the purposes of visual 
perception: first, we use vision to recognize objects and, 
secondly, we use vision for navigation and tracking. The 
purposes of imagery neatly parallel these purposes of vision: 
First, “recognition” is used in imagery as a way of recalling 
visual information. That is, imagery is used when one tries 
to answer questions from memory about subtle visual prop- 
erties of objects, resulting in one’s imagining an object and 
“recognizing” previously-unconsidered characteristics. For 
example, imagery tends to be used when one tries to answer 
questions like: “what shape are a beagle’s ears?” “which is 
larger, a goat or a hog?” or “which is higher off the ground, 
a race horse’s knees or the tip of its tail?” One is “recycling” 
stored visual memories, internally “looking” at the object 
again and recognizing previously-unconsidered properties. 
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Second, mechanisms used in navigation and tracking also 
may be used in the service of visual thinking. That is, one 
can mentally “move” imaged objects, or can “move” the self 
in relation to imaged objects, to “see” what would happen 
in the corresponding actual situation. For example, one 
might want to anticipate whether a vehicle will fit in a space 
between two trees, and so might project an image of the 
vehicle forward to “see” if it will fit; or one might want to 
know whether a tree would bridge a stream if it were 
chopped down and made to fall in a certain direction, and so 
might image the tree falling and “see” whether it would 
cross the stream. 

There is much empirical data indicating that imagery 
and like-modality perception share common mechanisms. 
For example, if one is holding a visual image (e.g., of a 
flower), this will impair visual perception more than it im- 
pairs auditory perception, but vice versa if one is holding an 
auditory image (e.g., the sound of a telephone ringing‘). 
Perhaps most interesting, manipulating objects in images 
reveals time-courses like those observed in the real world. 
For example, Figure | illustrates pairs of stimuli used in a 
classic study by Shepard and Metzler.° They asked subjects 
to decide if the objects were the same or different shape, 
irrespective of their orientation. Figure 2 presents the re- 
sults, indicating a highly linear increase in decision time as 


more mental rotation was required to bring the forms into 
congruence. This result is impressive because images are 
not actual, rigid objects and hence are not constrained by 
physics to have to pass through intermediate positions when 
the orientation of an imaged object is changed. Similar 
results are obtained with image scanning. Kosslyn, Ball and 
Reiser asked subjects to close their eyes and imagine the 
map illustrated in Figure 3.° This map had seven locations, 
which were positioned such that there were 21 distinct inter- 
location distances between all possible pairs. The subjects 
began by “focusing” on a given location on the imagined 
map (e.g., the tree), and then decided whether a second 
named location was or was not on the map (e.g., the hut 
versus a bench); they were asked to respond in the affirma- 
tive only after they had the second object clearly in focus. 
As is evident in Figure 4, increasingly more time was re- 
quired to scan between pairs of locations that were increas- 
ingly farther apart on the map, indicating that an imaged 
map can “stand in” for the actual one. Kosslyn’ reports 
another finding that is especially suggestive: If an object is 
imagined at a small size, more time is required to “see” its 
parts than if the object is imagined at a larger size.*-? This 
result is intriguing because it suggests that objects in images 
are subject to spatial summation, a well-known property of 
neural mechanisms used in vision. 





Figure 1: 


Stimuli used in the Shepard and Metzler experiment on 
mental rotation.> The stimuli in pairs 1 and 2 are congru- 
ent when rotated appropriately; the stimuli in pair 3 cannot 
be rotated into congruence. 
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Results from the Shepard and Metzler experiment on men- 
tal rotation.> Time to compare the shapes of forms in- 
creases linearly with the angular disparity between the 
forms. Rotation can be carried out both in the picture 
plane and in depth. 
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Figure 3: 


The fictional map used in the Kosslyn, Ball and Reiser ex- 
periment on image scanning.® The map was constructed 
so that there were 21 distinct distances between all possi- 
ble pairs of seven locations. 








Figure 4: 


Results from the Kosslyn, Ball and Reiser experiment on 
image scanning.® Time to scan across the imaged map 
increased linearly with the distance scanned. 
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Three Problems in Vision 

Although the behavioral phenomena are very interest- 
ing, they do not serve to specify a precise theory of how the 
brain is forming and using mental images. However, such 
data become very useful when we consider them in combi- 
nation with neuropsychological findings and computational 
theorizing. It is at this point that one of Marr’s ideas be- 
comes relevant. We can now consider an example of what 
Marr calls a “theory of the computation.”? A computation 
can be regarded as a “black box” that transforms input in a 
systematic way. A theory of a computation specifies what 
must be computed and why. Such a theory justifies positing 
a given computation by an analysis of what problems must 
be solved and the requirements on the solution to the prob- 
lems. That is, the goal of a computation is specified, as well 
as the nature of the input and constraints on the solution. 
This sort of theory is to be distinguished from a “theory of 
the algorithm,” which specifies the specific steps actually 
used to carry out a computation. The theory of the computa- 
tion is a first step, outlining the basic “processing compo- 
nents” that should be included in the theory; the theory of 
the algorithm fleshes out the details of how the computations 
are performed. In this paper, we will concentrate on the first 
level, focusing on a theory of the computations used in one 
aspect of imagery. 
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We can use the inference that imagery shares mecha- 
nisms with perception to discover a remarkable amount 
about the structure of the information processing system 
underlying imagery. We do so by first considering some 
fundamental problems that must be solved by a visual sys- 
tem. Our brains have apparently solved these problems in 
specific ways, and the outlines of these solutions are now 
apparent in the literature on the neurophysiology and neu- 
roanatomy of visual perception; these solutions have direct 
implications for a theory of visual processing. 

Thus, in this section we will begin to develop theories 
of some of the computations that are performed by the visual 
system. We will do so by considering three problems (which 
must be solved by any visual system) and the apparent 
solutions to these problems adopted by primate brains. In 
the following section, we will explore the implications of 
these inferences for a theory of imagery, assuming that 
visual imagery makes use of visual processing mechanisms. 


1. The Problem of Position Variability 

The same object is likely to occur at various positions 
in the visual field. Nevertheless, once we have seen an 
object, we can recognize it just as easily when it subse- 
quently is in a different position in the field. Logically, there 
are two ways we could perform this feat: On the one hand, 
when an object is encoded initially, the visual system could 
associate a separate representation with each of the possible 
positions of the object. This seems to be the choice made by 
McClelland and Rumelhart in their theory of word percep- 
tion.!° They associate a representation of each letter with 
each position in the field. (This was done so that the same 
letter could be detected in more than one position in a word.) 
On the other hand, when an object is encoded initially, it 
could be stored using representations that are associated 
with a set of positions in the field. In the limit, only one 
representation would be used for all positions. This is the 
solution Marr offered for the position variability problem; 
Marr suggested that the appearance of objects is stored in 
“object-centered” representations.’ In such representations, 
the locations of parts of objects are specified relative to other 
parts, not to positions in space. 

The solution adopted by primate visual systems to the 
problem of position variability is now evident in the neuro- 
physiological literature: It has been found in primates that 
visual cells in area TE (near the anterior end of the inferior 
temporal lobe) have very large receptive fields, and respond 
when patterns are present over a wide range of positions (the 
receptive field sizes are usually larger than 20 x 20 degrees 
of visual angle). This area of the brain has been shown to be 
critically involved in recognition per se.'' Thus, the pri- 
mate’s solution to the position variability problem relies on 
not representing the position of a pattern in the high-level 
shape representation system. (Incidentally, this is a good 
example of how facts about the neurological underpinnings 
of behavior can have direct bearing on theories of cognition; 
this finding is a significant challenge to the McClelland and 
Rumelhart model.)!° One implication of this solution is that 
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only one shape can be recognized at a time (although we 
could rapdily switch back and forth between stimuli, only 
one would be processed at any given instant); if multiple 
stimuli were being processed simultaneously, the large re- 
ceptive fields would often result in the system’s not being 
able to tell if there is one stimulus or two of the same stimuli 
being presented in different locations (e.g., the letter A in a 
word). Hence, figure/ground segregation is necessary to 
isolate individual patterns before they can be processed fur- 
ther. If this is done, then duplicate patterns can be isolated 
and processed separately, preventing confusions about how 
many of a pattern are present in the field. 

However, we do know where an object is when we see 
it. Thus, there must be a separate representation of an ob- 
ject’s location, which implies two separate mechanisms— 
one to represent a shape independently of its position and 
one to represent its position. And in fact, Ungerleider anu 
Mishkin summarize evidence for “two cortical visual sys- 
tems.”!? Their claim is that the ventral system, running from 
OC (primary visual cortex) through TEO down to TE, is 
concerned with analyzing what an object is, whereas the 
dorsal system, running almost directly from circumstriate 
area OB to OA and then to PG (in the parietal lobe) is 
concerned with analyzing where an object is. Figure 5 illus- 
trates the relevant areas of the brain. 





—_— 
Figure 5: 
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Visual areas of a rhesus monkey; these areas directly cor- 
respond to analogous areas in other primate brains. 
Ungerleider and Mishkin described two cortical visual sys- 
tems; the “dorsal system” runs through area OA to area 
PG in the parietal lobe, whereas the “ventral system” runs 
through area OA to area TE in the inferior temporal 


lobe. "2:13 











Two sorts of data are relevant to Ungerleider and 
Mishkin’s claim. First, the neuroanatomy and physiology 
support this distinction. There are well-known neural con- 
nections running along both pathways, and the visual prop- 
erties of these areas have been well documented.'* The 
visual areas of the parietal lobe appear to have different 
properties than those of the ventral visual system (e.g., they 
include the fovea in their receptive fields less often). Sec- 
ond, behavioral evidence suggests that animals are severely 
impaired in their ability to learn to discriminate between 
patterns if the inferior temporal lobes are removed. How- 
ever this lesion does not disrupt their ability to learn loca- 
tions. On the other hand, if the parietal lobes are removed, 
animals are severely impaired in their ability to discriminate 
on the basis of location, although they retain the ability to 
discriminate between patterns (see Ungerleider and Miskin, 
1982a,b). 

This interesting design of the processing mechanisms 
leads to difficulties that must be overcome by the system, as 
is evident when we consider another problem of visual per- 
ception. 


2. The Problem of Figure/Ground Segregation. 

Before we can recognize an object, “figure” must be 
segregated from “ground”; one must somehow pick out re- 
gions that are likely to correspond to distinct objects. The 
magnitude of difficulty of this problem becomes evident if 
you look at a digitized representation of a picture, with 
numbers representing the intensity of light at each point; the 
objects are overwhelmed by differences in lighting, texture, 
and so on, and it is very difficult to pick them out. A figure 
must be selected on the basis of physical properties of the 
input, such as regions of homogeneous color or texture, or 
contiguous zero-crossings in the second derivative of the 
function relating intensity to position (which occur at the 
edges of objects).> That is, because one has not yet identi- 
fied the object (segregating its form from the background is 
a logical prerequisite to recognition), one can only use phys- 
ical parameters to parse figure from ground. There are nu- 
merous proposals in the computer vision literature for ways 
of Organizing input into regions likely to correspond to fig- 
ures. 

An interesting problem arises here because the visual 
system processes input at different spatial frequency band- 
widths.'> Higher spatial frequencies correspond to more 
light/dark alternations per degree of visual angle; thus, 
higher resolution is required to detect higher spatial frequen- 
cies. The system can be described as having a number of 
different “channels,” each differing in resolution. At aver- 
age viewing distances, the lowest spatial frequency channel 
produces an output that will often correspond to the general 
shape envelope of an object. But consider what will happen 
at higher spatial frequency channels: the same factors that 
result in the parse of the object from the background will 
result in parts of a single object (e.g., the arms, legs, and 
head of a person) being parsed from one another. That is, 
the system cannot “know” what is an object and what is a 


part of an object; it just organizes regions in the input on the 
basis of physical parameters of the input array. And here lies 
a difficulty: once parsed, the shape representation system 
“ignores” the location in the visual field of the stimulus. 
Thus, the representation of the shapes of the parts will not 
preserve their positions. But the arrangement of parts is 
important for many recognition tasks. The relations must be 
represented somehow. 

The most straightforward solution to this problem re- 
quires a minor revision to the Ungerleider and Mishkin 
theory. It seems clear that “what” and “where” are not so 
distinct conceptually: sometimes the spatial relations among 
the parts are critical for identifying the form; for example, 
the difference between a Q with a long tail and an O with a 
diagonal slash through it is a matter of where the diagonal 
is positioned. Rather than “where,” the dorsal system seems 
specialized for representing spatial relations, including 
those among parts of a single object. The relations among 
high-resolution representations of parts presumably are rep- 
resented the same way as are the spatial relations among 
separate objects in a scene. (However, note that the parts 
and their relations are also implicit in the low-resolution, 
low spatial frequency representation; for example a handle 
of a mug will be a bulge on the blob-like representation of 
the mug.) In the relevant experiments, animals have never 
been required to discriminate among patterns that differ only 
in the relations among parts; usually stimuli differ in terms 
of numerous features, and the relationships among them are 
not important (e.g., as is true for the square and plus sign 
used by Ungerleider and. Mishkin, 1982b, which can be 
discriminated between simply by looking in the center of the 
figure and seeing if there is a line). In short, it would appear 
that once figure is segregated from ground, regardless of 
whether “figure” is an object or a part of an object, the 
location of that figure is represented in the parietal lobes. 


3. The Problem of Non-rigid Transformations. 

We can gain some insight into the way spatial relations 
are represented by considering another problem that must be 
contended with by a visual system: namely, the problem that 
many objects are subject to a near-infinite number of trans- 
formations, and so may not look the same from instance to 
instance. For example, a human form can be configured in 
a huge number of different ways, crouching, arms raised, 
standing on one toe with the arms held out to the side, and 
so on. Similarly, lewers of the alphabet can occur in numer- 
ous fonts, which are not simply linear transformations of 
each other. We cannot store a separate representation of all 
the possible configurations of such objects, with the aim of 
being able to match input to a specific stored representa- 
tion—there are simply too many possible configurations, 
and one often may encounter configurations not previously 
seen. Thus, it is useful to have a representation that will be 
general across a wide range of transformations. Two kinds 
of attributes remain constant under such transformations: 
First, the individual parts remain the same; although some 
may be hidden depending on the configuration, no parts are 
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actually added to or deleted from the object. Second, the 
topological-like relations among parts remain constant 
under all of these transformations. Topological relations are 
more abstract than the precise relative position of two parts 
as they appear in any given case (i.e., the topographic rela- 
tions); they indicate which parts are connected to each other 
and which are contained within each other. For example, the 
topological relation between the arm and shoulder remains 
constant under all of the different positions the arm can take. 
However, literally topological relations are too weak; the 
relation of ears to the side of the head, or the thumb to a 
hand, are important and will remain constant under transfor- 
mations. Thus, some general categories of relations must be 
used, not the actual topographic appearances. 

This problem places requirements on what the dorsal 
system must do: this system must be able to derive a descrip- 
tion of relations which will remain true no matter how the 
object is configured. These descriptions themselves cannot 
be images; images are concrete, being representations of 
specific instances. Instead, the dorsal system must be able 
to make use of more abstract, “categorical” representations. 
Such representations capture general properties of a rela- 
tionship without specifying the details (e.g., “left of” with- 
out specifying how much or exactly what angle). 

Finally, it would seem necessary that at some point the 
representations of perceptual units and their relationships 
must come together. A possible locus of that nexus is the 
association cortex near Wernicke’s area (in the posterior, 
superior temporal lobe), which appears to be involved in 
semantic processing. This arrangement is somewhat awk- 
ward, in that the relations must be delivered in synchrony 
with the related units; if the inputs fall out of phase, one may 
make “illusory conjunctions.” That is, one may conjoin 
units using the wrong relations. Interestingly enough, Treis- 
man and Gelade report that humans will see just such illu- 
sory conjunctions wher they are pushed to perform well in 
a difficult task.'® 


Computations Used in Imagery 

Given the sorts of empirical findings discussed in the 
introduction to this paper, it seems safe to assume that visual 
mental images are formed on the basis of representations 
initially encoded during perception. If so, then we are in a 
position to exploit our analysis of object representation to 
formulate a theory of the computations used in imagery. 
Before continuing, it may be useful to summarize where we 
have arrived up to here. The visual system appears to have 
two classes of mechanisms. The “ventral system” forms 
representations of an individual part or the overall shape 
envelope. Because the receptive fields are so large, this 
system does not register the locations of parts; hence, only 
one part can be processed at a time (otherwise it would not 
be able to tell when two of the same part—such as the same 
letter repeated in a word—are present). The representation 
of shape used in the ventral system should be concrete, 
capturing the precise shape and surface details of the part or 
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object. This system does not processs relations among parts, 
except insofar as they are implicit in a low resolution repre- 
sentation of the entire object (a kind of blurry blob-like 
form). In contrast, we are led to assume that the “dorsal 
system” must be able to derive abstract representations of 
the spatial relations among parts or objects. The use of 
categorical representations of spatial relations (e.g., 
“connected to”) is especially appropriate for classes of ob- 
jects whose members are subject to non-rigid transforma- 
tions. In such cases, the parts can be arranged in a large 
number of topographical configurations. 


Image Generation 

A fundamental question about imagery is, how are 
mental images formed? That is, when you try to decide what 
shape a beagle’s ears are, you will probably visualize the 
dog’s head and “look” at its ears. Until you tried to provide 
the information, there was no image. How was the image 
created? In this section we will focus on one aspect of an 
overall theory of imagery, namely the theory of image gen- 
eration. Our discussion of the three problems in vision has 
direct implications for a theory of the computations used in 
image generation. In this section, we will draw out these 
implications, and then will turn to an empirical test of these 
ideas in the following section. 

Presumably, the stored visual representations used in 
recognition can also be used in imagery, being activated to 
form an image. These representations are “concrete”, con- 
taining enough information to allow one to reconstruct the 
actual appearance. Image generation requires a computation 
that activates representations stored in memory; we can call 
this the PICTURE computation. The purpose of this compu- 
tation is to make explicit the topographical layout of parts of 
an object, which allows one to note previously unnoticed 
spatial relations (when imagery is being used in memory 
retrieval) and to “see” what would happen if one were mov- 
ing through space or the object were manipulated (when 
imagery is being used for visual thinking). Thus, the PIC- 
TURE computation presumably produces a pattern of acti- 
vation in a “visual buffer”; this pattern of activation is an 
“image representation”. The “visual buffer” is a 
functionally-defined storage medium that probably corre- 
sponds to the joint operation of numerous topographically 
organized areas of cortex.!’ A pattern of activation in this 
structure makes explicit the local geometric relations among 
parts of an object. We assume that these visual parts of 
cortex also can be activated from stored information, result- 
ing in a mental image. This buffer is equivalent to the buffer 
that supports Marr’s “2 1/2 D sketch” in vision.* 

The PICTURE computation activates an image of a 
single stored unit, which corresponds either to a part or a 
low-resolution image of the entire object (provided it was 
encoded). Because the spatial relations among parts are 
represented as descriptions, and presumably stored in that 
form, other computations must be used if a multipart or 
detailed object is to be imaged. For one, we need a compu- 
tation that can access stored descriptions of relations and use 





them to juxtapose separate parts in the correct relative posi- 
tions in an image. Let us call this the PUT computation. Part 
of what this computation must do is look up and interpret a 
description of how parts are to be arranged. For example, in 
generating a detailed image of a car, it might look up “front 
wheel” and discover the location description “under front 
wheelwell.” 

According to our analysis of the problem of position 
variability, units are encoded sequentially. This analysis, in 
conjunction with our analysis of requirements on represent- 
ing spatial relations among parts of non-rigid objects, sug- 
gests that local part-by-part relations are likely to be stored 
for non-rigid objects (such as part | is “left of,” “above,” or 
“connected to” part 2). Local part-by-part spatial relations 
are easiest to encode if parts are encoded one after the other, 
and are likely to remain constant under transformations of 
the object. If so, then the stored descriptions of inter-part 
relations will specify how one part is positioned relative to 
another (previously encoded) part. Thus, in generating 
multi-part images one must know where a reference point 
(e.g., the “wheelwell,” for a car’s wheel) is before being 
able to position another part correctly in an image. In order 
to locate a reference point, a third computation, which we 
can call the FIND computation, must be used. The FIND 
computation takes as input a pattern in the visual buffer and 
produces a label for it (obviously, this computation is in fact 
a collection of sub-computations). It is used to locate the 
“foundation part” on an already-imaged part (i.e., where the 
new part belongs, “front wheelwell” in the example). In 
order to position a new part correctly, the PUT computation 
must use the output from the FIND computation (e.g., the 
location on the body where the front wheelwell belongs) 
plus the description of the relation (“under”) to compute 
parameter values for the PICTURE computation. These 
values are provided to the PICTURE computation, allowing 
the new part to be imaged in the correct relation to the 
foundation part. 


Brain Damage and Image Generation 
The theory of image generation developed here is a 
theory that posits a set of distinct computations carried out 
by the brain. One source of support for such a theory would 
be the finding of “functional dissociations” among the puta- 
tive components. That is, some type of brain damage may 
leave some of the components intact while disrupting the 
others. As a specific example, let us review recent evidence 
supporting the existence of a distinct PUT computation. 
Kosslyn, Holtzman, Farah, and Gazzaniga investi- 
gated a possible dissociation between the FIND and PIC- 
TURE computations on the one hand, and the PUT compu- 
tation on the other hand.'* These studies examined the 
isolated cerebral hemispheres of “split-brain” patients, who 
have had their corpus callosa—the major connection be- 
tween the hemispheres—surgically severed (for medical 
reasons). Because the PUT computation putatively involves 
operations also used in language (accessing and interpreting 


descriptions), we expected that it might typically be more 
effective in the left cerebral hemisphere, which is special- 
ized for language processing in most people (for a more 
precise theory-underlying this expectation, see Kosslyn ref- 
erence 19). In contrast, because both hemispheres can rec- 
ognize and remember shapes, we expected the PICTURE 
and FIND computations to be equally effective in both 
hemispheres. That is, the PICTURE computation simply 
activates stored visual representations, which exist in both 
hemispheres, and the FIND computation identifies patterns, 
which also is done in both hemispheres. Thus, we hoped to 
show that the isolated left hemisphere could perform image 
generation tasks that involve all three computations, 
whereas the right hemisphere could only perform image 
generation tasks that require the PICTURE and FIND com- 
putations but not the PUT computation. This prediction is 
especially interesting because the common wisdom has it 
that the right cerebral hemisphere is the seat of mental im- 
agery.”°.2! Thus, if we can show that the left hemisphere is 
selectively able to perform imagery tasks better than the 
right hemisphere, counter to the common sense notions 
about imagery and the right hemisphere, this demonstration 
will be good evidence of the usefulness of the computational 
approach. In addition, in the course of implicating a specific 
processing deficit in the right hemisphere, we will see the 
usefulness of cognitive methods applied to the investigation 
of neuropsychological phenomena. 

Before beginning the investigation proper, preliminary 
experiments were conducted with normal subjects to show 
that people actually use imagery to perform the tasks. The 
task of most interest simply required subjects to decide from 
memory if upper case letters of the alphabet are composed 
only of straight lines or include some curved lines. The 
upshot of the preliminary work was that imagery is indeed 
used in this task, and that images of upper case letters are 
usually generated a part at a time—which putatively re- 
quires descriptive relations among the parts. The weak link 
here, of course, is the assumption that because letters are 
imaged a part at a time, they must be imaged on the basis 
of a stored description. This claim is reasonable in light of 
the fact that letter fonts differ widely, and hence letters are 
objects subject to non-rigid transformations. If so, then one 
typically will not store representations of particular letters 
(e.g., a B seen on page 4 of yesterday’s Times), but will 
store an abstract description of the relations among the parts, 
and this description later be used to form an image. 


An Image Generation Deficit 

The visual system is constructed conveniently for the 
scientist interested in presenting a stimulus to only a single 
hemisphere of the brain. As is illustrated in Figure 6, the 
right half of each retina projects only to the right cerebral 
hemisphere, and the left half of each retina projects only the 
left cerebral hemisphere. Thus, if a subject is asked to stare 
at a point directly in front of him or her, a stimulus flashed 
to the right side will strike only the left half of each retina— 
resulting in its being projected to only the left hemisphere; 


Four / 1985 37 





similarly, a stimulus flashed to the left side will project only 
to the right hemisphere. If the stimulus is presented for 
under about 200 msec, the subject will be unable to move 
his or her eyes before it is gone (an eye movement requires 
about 200 msec to perform), ensuring that the stimulus falls 
only one half of each retina. 





Figure 6: 


A schematic top view of the visual system illustrating how 
the left half of each retina projects only to the left cerebral 
hemisphere and the right half of each retina projects only 
to the right cerebral hemisphere. 








To get a sense of how computational theories can be 
tested using cognitive neuropsychological methodologies, 
we will focus on a series of experiments done with split- 
brain patient, J.W. This patient had his corpus callosum 
sectioned about three years prior to testing, as a treatment 
for severe intractable epilepsy; he has been extensively 
tested and it has been found that his right hemisphere is 
capable of comprehending involved verbal instructions and 
of making simple deductions and classifications.” 

In the first experiment, J.W. was asked to look straight 
ahead, and read the lower case letters flashed to either side. 
These letters were exposed for only 150 msec, ensuring that 
only one hemisphere saw them. J.W. was to decide whether 
or not the corresponding upper case version of each letter 
had any curved lines. He pressed one button if he thought 
the upper case letter had curves, and another if he thought 
it only had straight lines. The left arm was used for all 
responding (due to ipsilateral efferents, both hemispheres 
have some control over the major arm movements). 

The results of the first sets of trials were straightfor- 
ward: J.W. performed almost perfectly in his left hemi- 
sphere, and showed severely degraded performance in his 
right hemisphere. 
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A task analysis suggests that this deficit could be due 
to problems in carrying out any one or combination of seven 
processing stages. That is, to peform this task successfully 
one must: ENCODE the cue letter, ACCESS the upper case 
representation, GENERATE the image, RETAIN the image 
long enough to use it, INSPECT the image, EVALUATE 
the shape, and RESPOND. Furthermore, the deficit could 
be due to the right hemisphere’s not understanding the in- 
structions and/or to its not being able to combine stages. 
Thus, we had to show that the right hemisphere’s deficit 
could be ascribed to difficulty in generating the image per 
se, and then we had to try to isolate which image generation 
computation or computations was deficient. 

A number of control experiments were conducted to 
implicate a deficit in image generation per se in the right 
hemisphere. For example, in one control experiment the 
upper case letters themselves were presented, and J.W. was 
asked to classify what he saw; both hemispheres performed 
virtually perfectly. In another experiment, 3-letter words 
composed of upper case letters (e.g., MUG) were presented 
to one hemisphere or the other, and J.W. was cued as to 
which letter (first, second, or third) to classify as being 
straight or curved. The cue was given either two seconds 
before the word was presented, making it a perceptual clas- 
sification task, or two seconds after the word was presented, 
requiring him to retain and classify an image. There was no 
difference in how well each hemisphere performed in the 
imagery and perceptual conditions. Thus, the problem was 
not that the right hemisphere a) could not ENCODE the 
letters, b) could not RETAIN an image, c) could not IN- 
SPECT an image, d) could not make the straight/curved 
EVALUATION, e) could not RESPOND, of f) could not 
understand the instructions. In addition the word-task re- 
quired combining these stages together. In another task, it 
was found that the right hemisphere could use a lower case 
letter cue to ACCESS the other-case representation: when 
shown the lower case letter, both hemispheres could pick 
out the corresponding upper case version. Numerous other 
control experiments were conducted to provide convergent 
evidence for the inference that all stages but image genera- 
tion were intact in the right hemisphere of this patient. !% 


Implicating a Computation 

The results described so far only serve to demonstrate 
an image generation deficit in J.W.’s right hemisphere. 
They do not implicate a deficit in one computation or an- 
other. In order to implicate the PUT computation it must be 
shown that both hemispheres can use the PICTURE and 
FIND computations. These are the only computations re- 
quired to generate images of a single part or overall shape; 
when multiple parts are composed, the PUT computation is 
putatively required. If so, then J.W.’s right hemisphere 
should be able to form an image of the general shape enve- 
lope of an object, which is presumed to be encoded as a 
single perceptual unit at a low spatial frequency channel (as 
a degraded blob-like form, similar to the appearance of a 
blurred slide). And indeed, it was demonstrated that both of 





J.W.’s hemispheres could perform tasks requiring the imag- 
ing of general shapes: In one experiment, the names of 
animals were presented to one hemisphere or the other, and 
J.W. was asked whether that animal was larger than a goat 
(pressing one button to indicate yes and another to indicate 
no). The named animals were of very similar sizes to a goat 
(e.g., hog, wolf), and this task previously had been shown 
to require imagery in normal subjects (see chapter 9 of 
reference 8). Thus, it is of interest that both of J.W.’s 
hemispheres performed this task virtually perfectly. In an- 
other experiment, it was found that J.W. also could decide 
equally well in both hemispheres whether a named object 
(e.g., a cup) was higher than it is wide. These tasks require 
the PICTURE computation to generate images of the gen- 
eral shapes of the objects, and the FIND computation to 
inspect the objects in the image. Because no parts need to be 
added to the general shape, the PUT computation is not 
necessary. 

To make the case that there is a deficit in one particular 
computation in J.W.’s right hemisphere, we must consider 
one more result. So far the results demonstrate that J.W.’s 
right hemisphere cannot generate images of upper case let- 
ters, but can image and compare object shapes. Perhaps the 
problem is in generating images of letters per se. To elimi- 
nate this possibility, the animal names used in the size 
judgment experiment were again presented separately to 
each hemisphere. Now, however, J.W. was to decide 
whether each animal’s ears protrude above the top of the 
skull or not (e.g., horse versus sheep). This task requires 
imaging the ears in the correct relative location to the head. 
In this task, J.W.’s right hemisphere performed at chance— 
in contrast to the virtually perfect performance exhibited by 
the left hemisphere. This was true even though the right 
hemisphere judged the same physical stimuli (animal 
names) virtually perfectly in the size comparison task. Thus, 
it appears that the right hemisphere has selective difficulty 
when parts must be arranged correctly in an image. 

These experiments have also been conducted on an- 
other split-brain patient, V.P., which also produced evi- 
dence for a right-hemisphere deficit in image generation. 
Unlike J.W., however, V.P. eventually learned to generate 
images of letters in her right hemisphere—but only for the 
specific letters she had previously practiced imaging. It is of 
interest that unlike J.W., whose corpus callosum is cleanly 
severed, parts of the splenium were inadvertently spared in 
V.P. (as was recently discovered through NMR scans). The 
splenium is part of the callosum known to transfer visual 
information between the hemispheres. In addition, similar 
results have been obtained with normal subjects when we 
examine the time to rsspond depending on which hemi- 
sphere sees the cue first. 

The results of these experiments, then, documented a 
highly selective deficit in the right hemisphere for one com- 
putation involved in image generation. The computational 
theory proved its worth by leading us to investigate a 
counter-intuitive claim that probably would not have arisen 
using traditional approaches. 


Conclusions 


Perhaps the greatest indicator of success in science is 
the discovery of underlying structure where none was orig- 
inally apparent. By this measure, the present approach has 
promise. A computational analysis of imagery and vision 
led to a novel analysis of the structure of one imagery 
ability, the ability to form visual mental images. In addition, 
cognitive methods applied to neuropsychological phenom- 
ena allowed us to test and verify a non-intuitive implication 
of the theory, namely that the left cerebral hemisphere 
would be better than the right at specific imagery tasks. 

This approach not only has promise of helping us to 
discover the underlying mechanisms of mental imagery, but 
may help us to understand the functional organization of 
“mental” processes in the brain more generally. The notion 
that the left and right hemispheres have distinct functions is 
not new, and in fact has penetrated deeply into the popular 
consciousness. Indeed, many people attempt to characterize 
themselves as predominently “left-” or “right-brained,” as- 
suming that the hemispheres differ along some simple di- 
chotomy (e.g., “verbal” versus “imaginal”). If the present 
approach is correct, these dichotomies are hopelessly over- 
simplified and have little utility (e.g., both hemispheres are 
imaginal, but in different ways). A more refined theory will 
specify the components of processing that are better devel- 
oped in one hemisphere relative to the other, and will 
specify how these components are brought to bear in helping 
one to perform specific tasks. 

Detailed theories of the computations used in process- 
ing will be very complicated and, without question, com- 
puter simulation will be required to develop them and ex- 
plore their implications. That is, we will have to embody the 
putative computations in a computer model, and observe its 
behavior in various tasks. This conclusion is particularly 
evident when we compare the results we have obtained for 
different subjects in the experiments described above, which 
vary slightly from person to person. Presumably, different 
computations are more or less effective for different people, 
and any detailed simulation will have to account for this by 
specific parameter variations. Fortunately, the technology is 
now available for just such future developments. 
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Research Notes 


Establishment of a New Center for 
Brain Research 


New York University Medical Center and Biomagnetic 
Technologies, Inc. have announced a collaborative research 
program in magnetoencephalography (MEG). The basic re- 
search underlying this venture has been supported by the 
Office of Naval Research principally at NYU and the Uni- 
versity of California, Los Angeles. (The original research 
used a single channel second order gradiometer). A 100 
channel device is now being built for delivery in about two 
years. Neuromagnetometry offers a non-invasive way to 
monitor human brain activity more precisely than previously 
possible. The early recognition by ONR of the potential of 
the MEG technique and the continued support over the past 
ten years is directly responsible for this latest development. 


(D. P. Woodward, ONR) 


Installation of a Multiple Channel 
Neutromagnetometer at UCLA 


The world’s second five channel DC neuromagnetome- 
ter was delivered to the Human Neurophysiology Labora- 
tory at the University of California, Los Angeles. This de- 
vice will greatly speed the neuromagnetic investigations of 
cognitive functions by Jackson Beatty and his coworkers. 
This project has been sponsored by the Physiology Program 
of the Office of Naval Reserach. 

The instrument itself is somewhat more advanced than 
the original multichannel system, both of which were man- 
ufactured by Biomagnetic Technologies, Inc., of San 
Diego. The UCLA system has an improved sigiial to noise 
ratio, made possible in large part by recent advances in 
shielding technologies. Further, the UCLA laboratory is a 
magnetically quiet site. The combination of low instrument 
and enviromental noise levels permits operation of the neu- 
romagnetormeter at unusually high grain levels. The result 
is a major enhancement in total measurement sensitivity. 

The five recording channels are configured as second 
derivative gradiometers with extended baselines for im- 
proved sensitivity to deep cranial signals. Coil diameter is 
1.5 cm with 2 cm inter-coil spacing. An additional four 
magnetometer channels provide for adaptive noise cancella- 
tion. 

The instrument will be used in a program of research 
designed to locate and measure intracranial currents mediat- 
ing early vision, linguistic processing, and attention. Neuro- 
magnetic imaging provides a unique view of cortical func- 
tion, permitting both three dimensional spatial resolution 
and millisecond temporal resolution. The technique has al- 
ready proven to be a powerful new tool for brain research; 
the multichannel system represents an improvement of mea- 
surement technology that will speed experimental research 
by at least one order of magnitude. 


(D. Woodward, ONR) 


The Butterfly Computer 


BBN Laboratories’ Butterfly™ parallel processing 
system achieves nearly linear speedup with 128 processors, 
for a total of 60 million instructions per second (MIPS). The 
128-processor Butterfly system is configured in 4 racks. 
Each processor node consists of an 8 MHz MC68000 pro- 
cessor, a CO-processor, memory management, switch inter- 
face and I/O interface. 

Parallel computers, such as the Butterfly, are playing 
an increasingly important role in Computer Science research 
and practice. It has been established that powerful parallel 
computers can be built and programmed to be extremely 
cost-effective for certain problems. The challenge is to ex- 
tend the range of tractable problems and to provide program- 
ming languages and environments that support parallel com- 
puter, and several research groups are finding fruitful 
interactions among studies of natural and artificial computa- 
tion. 

One of these interdisciplinary efforts is at the Univer- 
sity of Rochester, which will install the second 128- 
processor Butterfly computer later this year. In a project 
jointly sponsored by Office of Naval Reserach, National 
Science Foundation, and Defense Advanced Research 
Projects Agency, the Rochester group will be studying both 
conventional programming and “massively parallel” com- 
putational models for the Butterfly. The massively parallel 
approach attempts to decompose problems into computa- 
tions done by millions or billions of simple units. This 
approach has been effective for a variety of tasks and the 
obvious connection to neurons is the basis for interdisci- 
plinary research, some of which is described in this issue. 


(Joel Davis, ONR) 


Butterfly Computer of BBN Laboratories, Cambridge, Massachusetts. 
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